Information processing apparatus, information processing method, program, and moving body

ABSTRACT

The present technique relates to an information processing apparatus, an information processing method, a program, and a moving body that can appropriately display content on top of a scene viewed by a user. 
     An aspect of the present technique provides an information processing apparatus that sets a frame as a superimposition location of content in a region corresponding to a surface of an object on the basis of a movement state of a user and generates visual information for displaying the content in the region corresponding to the set frame. The present technique can be applied to an apparatus that performs AR display of content.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.16/640,727, filed Feb. 21, 2020, which is based on PCT filingPCT/JP2018/030489, filed Aug. 17, 2018, which claims priority to JP2017-166871, filed Aug. 31, 2017, and JP 2018-069055, filed Mar. 30,2018, the entire contents of each are incorporated herein by reference.

TECHNICAL FIELD

The present technique relates to an information processing apparatus, aninformation processing method, a program, and a moving body, andparticularly, to an information processing apparatus, an informationprocessing method, a program, and a moving body that can appropriatelydisplay content on top of a scene viewed by a user.

BACKGROUND ART

There is a technique of using a head-up display to project, to awindshield, information regarding a facility or the like included in avisual field of a driver and presenting the information to the driver.The driver views various types of information on top of a scenespreading in front of the driver.

PTL 1 discloses a technique of displaying an AR (Augmented Reality)image regarding a facility in association with a real scene viewed bythe driver, in which the AR image is displayed far when the vehiclespeed is fast, and the AR image is displayed close when the vehiclespeed is slow.

CITATION LIST Patent Literature [PTL 1]

-   Japanese Patent Laid-Open No. 2015-77876

SUMMARY Technical Problem

It is desirable to present information in a form that the informationcan be viewed as if the information is fit into the scene. By displayingthe information such that the information is fit into the scene, thedriver can view the information while concentrating on the drive.

The present technique has been made in view of the circumstances, andthe present technique can appropriately display content on top of ascene viewed by a user.

Solution to Problem

An aspect of the present technique provides an information processingapparatus including a setting unit that sets a frame as asuperimposition location of content in a region corresponding to asurface of an object on the basis of a movement state of a user, and adisplay control unit that generates visual information for displayingthe content in the region corresponding to the set frame.

In the aspect of the present technique, the frame as the superimpositionlocation of the content is set in the region corresponding to thesurface of the object on the basis of the movement state of the user,and the visual information for displaying the content is generated inthe region corresponding to the set frame.

Advantageous Effect of Invention

According to the present technique, the content can be appropriatelydisplayed on top of the scene viewed by the user.

Note that the advantageous effect described here may not necessarily belimited, and the advantageous effect may be any of the advantageouseffects described in the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an interior of a vehicle provided witha vehicle control system according to a first embodiment of the presenttechnique.

FIG. 2 is a diagram illustrating an example of an appearance of content.

FIG. 3 depicts diagrams each illustrating a display example of thecontent for realizing the appearance illustrated in FIG. 2 .

FIG. 4 is a diagram schematically illustrating display of the content.

FIG. 5 is a diagram illustrating an example of the appearance of thecontent after a lapse of certain time from the state of FIG. 2 .

FIG. 6 is a diagram illustrating an example of the appearance of thecontent after a lapse of certain time from the state of FIG. 5 .

FIG. 7 is a diagram illustrating an example of the appearance of thecontent after a lapse of certain time from the state of FIG. 6 .

FIG. 8 is a block diagram illustrating a configuration example of thevehicle control system of the first embodiment.

FIG. 9 is a block diagram illustrating a functional configurationexample of an output control unit of FIG. 8 .

FIG. 10 is a block diagram illustrating a configuration example of aninformation superimposition appropriate visual field setting unit ofFIG. 9 .

FIG. 11 is a diagram illustrating an example of a three-dimensionalmodel of an object.

FIG. 12 is a diagram illustrating an example of setting informationsuperimposition possible frames.

FIG. 13 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by an exclusion rule 1.

FIG. 14 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by an exclusion rule 2.

FIG. 15 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by an exclusion rule 3.

FIG. 16 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by an exclusion rule 4.

FIG. 17 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by an exclusion rule 5.

FIG. 18 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by an exclusion rule 6.

FIG. 19 is a diagram illustrating an example of transition of a vehicleposition.

FIG. 20 is a diagram illustrating an example of a change in direction ofthe vehicle.

FIG. 21 depicts diagrams illustrating scene images at vehicle positions.

FIG. 22 is a diagram illustrating an example of an angular velocity in auser visual field.

FIG. 23 is a diagram illustrating an example of an angle with respect toa vehicle traveling direction.

FIG. 24 is a diagram illustrating an example of an exposed area ratio.

FIG. 25 is a diagram illustrating the example of the exposed area ratio.

FIG. 26 is a diagram illustrating an example of stay time in the uservisual field.

FIG. 27 is a diagram illustrating an example of setting an informationsuperimposition appropriate visual field.

FIG. 28 is a block diagram illustrating a configuration example of asuperimposition target frame selection unit of FIG. 9 .

FIG. 29 depicts diagrams each illustrating an example of contentinformation items.

FIG. 30 depicts diagrams each illustrating an example of fitness.

FIG. 31 is a diagram illustrating an example of setting superimpositionlocations.

FIG. 32 is a diagram illustrating an example of the superimpositionlocations.

FIG. 33 is a block diagram illustrating a configuration example of adisplay control unit of FIG. 9 .

FIG. 34 is a diagram illustrating an example of adjustment of contraston the basis of fitting contrast.

FIG. 35 is a diagram illustrating an example of display of a frontobject.

FIG. 36 is a flow chart describing an information display process.

FIG. 37 is a flow chart describing an information superimpositionappropriate visual field setting process executed in step S3 of FIG. 36.

FIG. 38 is a flow chart following FIG. 37 , describing the informationsuperimposition appropriate visual field setting process executed instep S3 of FIG. 36 .

FIG. 39 is a flow chart describing a superimposition target frameselection process executed in step S4 of FIG. 36 .

FIG. 40 is a flow chart describing a display process executed in step S5of FIG. 36 .

FIG. 41 is a flow chart describing a pre-analysis process of content.

FIG. 42 is a diagram describing a summary of an example ofsuperimposing, defocusing, and displaying the content on informationsuperimposition appropriate frames according to a visual point positionof the user when there is one user according to a second embodiment ofthe present technique.

FIG. 43 is a diagram describing a setting example of a defocus range ina case where there is one information superimposition appropriate frame.

FIG. 44 is a diagram describing a setting example of defocus ranges in acase where there is a plurality of information superimpositionappropriate frames and describing a display example when a projectionunit is a transmissive display.

FIG. 45 is a diagram describing a configuration example of the displaycontrol unit according to the second embodiment of the presenttechnique.

FIG. 46 is a diagram describing a configuration example of a displayprocessing unit of FIG. 45 .

FIG. 47 is a flow chart describing a display process of the displaycontrol unit of FIG. 45 .

FIG. 48 is a flow chart describing a defocus range map generationprocess of step S202 in FIG. 47 .

FIG. 49 is a diagram describing a configuration example of the displayprocessing unit when the projection unit is a non-transmissive displayaccording to a modification of the second embodiment of the presenttechnique.

FIG. 50 is a flow chart describing a display process of the displaycontrol unit of FIG. 45 in which the display processing unit of FIG. 49is applied.

FIG. 51 is a diagram describing a setting example of the defocus rangesin a case where there is a plurality of information superimpositionappropriate frames and describing a display example when the projectionunit is a non-transmissive display.

FIG. 52 is a diagram describing a summary of an example ofsuperimposing, defocusing, and displaying the content on the informationsuperimposition appropriate frames according to the visual pointpositions of a plurality of users when there is a plurality of usersaccording to a third embodiment of the present technique.

FIG. 53 is a diagram describing a configuration example of a displayunit using a liquid crystal polarization shutter in the case ofsuperimposing, defocusing, and displaying the content on the informationsuperimposition appropriate frames according to the visual pointpositions of a plurality of users when there is a plurality of users.

FIG. 54 is a block diagram illustrating a configuration example of thevehicle control system of the third embodiment.

FIG. 55 is a block diagram illustrating a configuration example of thedisplay processing unit when the liquid crystal polarization shutter isused for the display unit in the display control unit of the outputcontrol unit in the vehicle control system of FIG. 54 .

FIG. 56 is a flow chart describing a display process of the displaycontrol unit of FIG. 45 in which the display processing unit of FIG. 55is applied.

FIG. 57 is a diagram describing a configuration example of the displayunit using lenticular lenses in the case of superimposing, defocusing,and displaying the content on the information superimpositionappropriate frames according to the visual point positions of aplurality of users when there is a plurality of users.

FIG. 58 is a block diagram illustrating a configuration example of thedisplay processing unit when the lenticular lenses are used for thedisplay unit in the display control unit of the output control unit inthe vehicle control system of FIG. 54 .

FIG. 59 is a flow chart describing a display process of the displaycontrol unit of FIG. 45 in which the display processing unit of FIG. 58is applied.

FIG. 60 is a block diagram illustrating a configuration example of acomputer.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present technique will be described. Theembodiments will be described in the following order.

1. First Embodiment 1-1. Display Example of Content 1-2. ConfigurationExample of Vehicle Control System 1-3. Motion of Vehicle Control System1-4. Modification 2. Second Embodiment 2-1. Summary of Second Embodiment2-2. Setting Example of Defocus Range 2-3. Setting Example of DefocusRanges in Case Where There Is Plurality Of Information SuperimpositionAppropriate Frames 2-4. Configuration Example of Second Embodiment ofDisplay Control Unit 2-5. Modification 3. Third Embodiment 3-1. Summaryof Third Embodiment 3-2. Configuration Example of Vehicle Control System3-3. Display Process of Display Processing Unit in FIG. 55 3-4.Modification 1. First Embodiment <<1-1. Display Example of Content>>

FIG. 1 is a diagram illustrating an interior near a driver's seat of avehicle provided with a vehicle control system according to anembodiment of the present technique.

As illustrated in FIG. 1 , a windshield G is provided beyond a dashboardas viewed from a user (driver) sitting on the driver's seat. The vehiclecontrol system displays content by, for example, projecting an image onthe windshield G. The user views the content on top of a scene that canbe viewed through the windshield G.

Note that, although the occupant of the vehicle viewing the content isthe driver in the case described below, the user viewing the content mayalso be another user sitting on the front seat or the back seat.

Examples of the content to be displayed include images representingvarious types of information, such as entertainment information,practical information, and advertisement. The images displayed ascontent are moving images in some cases and still images in other cases.The images may or may not include characters.

A projection unit that realizes AR display of the content is provided ona predetermined position, for example, on the back side of the rearviewmirror and the upper surface of the dashboard. Other than the projectionunit, various devices that project virtual images may be used as devicesthat realize the AR display. For example, a transmissive displayattached to the windshield G may be used, or a transmissive HMD (HeadMounted Display) worn by the user may be used.

The vehicle control system uses optimal expression to display thecontent at an optimal place in the scene. Specifically, among frames seton a wall surface of an object, such as a building, existing in thescene, a frame that can be easily viewed by the user is selected as theoptimal place on the basis of a movement state. In addition, to providethe content as if the content is actually projected on the wall surfaceof the building, the content is displayed after executing imageprocessing according to the appearance of the wall surface.

FIG. 2 is a diagram illustrating an example of the appearance of thecontent.

A scene depicted in a range of a horizontally long rectangle illustratedin FIG. 2 represents part of the scene viewed by the user sitting on thedriver's seat. A curve to the right can be viewed in front of thevehicle. There is a building B1 on the near side on the right of thecurve, and there is a building B2 on the far side of the curve. There isa building B3 further beyond the building B2 along the road.

As illustrated in FIG. 2 , a rectangular frame modified according to theorientation of the wall surface is set on the wall surface of thebuilding B1, and content C1 is displayed in the frame.

Similarly, content C2 is displayed in the frame set on the wall surfaceof the building B2, and content C3 is displayed in the frame set on thewall surface of the building B3. The content is displayed large on thewall surface of a building at a short distance, and the content isdisplayed small on the wall surface of a building at a long distance.

The vehicle speed is displayed on the upper left. In this way, varioustypes of information other than the content can also be displayed.

FIG. 3 depicts diagrams each illustrating a display example of thecontent for realizing the appearance of the scene as illustrated in FIG.2 .

A of FIG. 3 illustrates an actual scene viewed in front of the user. Asillustrated in A of FIG. 3 , the curve to the right can be viewed infront of the vehicle, and the buildings B1 to B3 can be viewed along theroad in the actual scene.

The contents C1 to C3 are displayed in the scene in a form asillustrated in B of FIG. 3 , and the same appearance of content as inFIG. 2 is realized as illustrated in C of FIG. 3 .

FIG. 4 is a diagram schematically illustrating the display of thecontent.

The content is displayed by displaying the image in a region on thewindshield G corresponding to the frame set on the building. On thebasis of a position P that is the position of the user, a region insidestraight lines connecting the position P and the frame (diagonal linepart) is the region on the windshield G corresponding to the frame. Ahorizontal line in FIG. 4 represents the windshield G as a displaysurface between the user and the building.

In the example of FIG. 4 , a region on the windshield G corresponding toa frame set on a building B1′ is a region G1, and a region on thewindshield G corresponding to a frame set on a building B2′ is a regionG2. The content can be displayed in each of the region G1 and the regionG2 to realize the appearance as if the content is displayed on each ofthe wall surfaces of the building B1′ and the building B2′.

The user views the content displayed in the region G1 on top of thebuilding B1′ viewed through the windshield G. In addition, the userviews the content displayed in the region G2 on top of the building B2′viewed through the windshield G. In this way, the user can view varioustypes of information as if the information is displayed on the wallsurfaces of the buildings.

In this way, the superimposition of the content on the frames set on thewall surfaces of the buildings is realized by displaying the images onthe windshield G.

The displayed content is updated in real time according to the travelingstate (movement state) during the traveling of the vehicle. Theappearance of the content also changes with a change in appearance ofthe scene from the driver's seat.

FIG. 5 is a diagram illustrating an example of the appearance of thecontent after a lapse of certain time from the state of FIG. 2 .

The state of FIG. 5 is a state in which the vehicle is turning thecurve. Most of the building B1 viewed on the right is out of the visualfield, and the content C1 superimposed on the building B1 cannot beviewed.

In addition, the building B2 is getting closer in front of vehicle, andthe content C2 is displayed larger than in the state of FIG. 2 . Thecontent C2 is displayed in a size and a shape as if the content C2 isdisplayed on the wall surface of the building B2 on the basis of thepositional relationship between the vehicle and the wall surface of thebuilding B2. The details of the displayed content C2 also change with alapse of time.

Similarly, the display of the content C3 also changes with a change inappearance of the building B3.

In addition, a building B4 not viewed in the state of FIG. 2 is newlyviewed beyond the building B3 in the example of FIG. 5 . Frames are alsoset on two wall surfaces of the building B4 that can be viewed from theuser, and content C4-1 and content C4-2 are superimposed on the frames,respectively.

FIG. 6 is a diagram illustrating an example of the appearance of thecontent after a lapse of certain time from the state of FIG. 5 .

The state of FIG. 6 is a state in which the vehicle is moving straightthe road after turning the curve. Most of the building B2 viewed on theleft is out of the visual field, and the content C2 superimposed on thebuilding B2 cannot be viewed.

Furthermore, in addition to a building B5 hardly viewed in the state ofFIG. 5 , buildings B6 and B7 can be viewed in front, and content issuperimposed on each of the wall surfaces.

FIG. 7 is a diagram illustrating an example of the appearance of thecontent after a lapse of certain time from the state of FIG. 6 .

The state of FIG. 7 is a state in which the vehicle is moving straightat a position in front of the building B4. The display of the contentsuperimposed on each of the buildings B4 to B7 is changed with a changein appearance of the building. In addition, a building B8 hardly viewedin the state of FIG. 6 can be viewed large on the right, and content isalso superimposed on the wall surface of the building B8.

Note that, although the frames are set on the wall surfaces of thebuildings in the case described here, the frames are also set on variousstructures, such as a water storage tank and sound-proof walls installedon both sides of a highway, and the content is superimposed. The targetof setting the frames is not limited to the structures, and the targetmay also be natural objects, such as a slope of a mountain viewed at along distance and a sea surface. The frames may also be set on the roaditself, and the content may be superimposed.

That is, the frames can be set on various objects in the visual field ofthe user driving the vehicle, and the content can be superimposed. Inthe following description, the object as a superimposition location ofthe content is mainly the building.

In this way, the vehicle control system displays the content bysuperimposing the content on the building viewed during driving.Furthermore, the content superimposed on the building is displayed bychanging the shape and the size according to the change in appearanceviewed by the user.

As described in detail later, image processing is actually applied tothe content by reducing the contrast in a case where the building is ata long distance so that the building looks blurry or by reducing thecontrast in a case where intense light shines on the building so thatthe building looks bright.

The content is displayed in a form as if the content is displayed on thebuilding, and the display changes with a change in appearance of thescene. Therefore, the user can view the content in a natural form whiledriving.

A process of the vehicle control system that realizes the display of thecontent will be described later with reference to flow charts.

<<1-2. Configuration Example of Vehicle Control System>> <1-2-1. OverallConfiguration of Vehicle Control System>

FIG. 8 is a block diagram illustrating a configuration example of avehicle control system 100. The vehicle control system 100 illustratedin FIG. 8 functions as an information processing apparatus.

Note that, in a case where the vehicle provided with the vehicle controlsystem 100 is to be distinguished from other vehicles, the vehicle willbe appropriately referred to as a user's car or a user's vehicle.

The vehicle control system 100 includes an input unit 101, a dataacquisition unit 102, a communication unit 103, a vehicle interiordevice 104, an output control unit 105, an output unit 106, a drivecontrol unit 107, a drive system 108, a body control unit 109, a bodysystem 110, a storage unit 111, and an automatic drive control unit 112.The input unit 101, the data acquisition unit 102, the communicationunit 103, the output control unit 105, the drive control unit 107, thebody control unit 109, the storage unit 111, and the automatic drivecontrol unit 112 are connected to each other through a communicationnetwork 121.

The communication network 121 includes, for example, an on-boardcommunication network in compliance with an arbitrary standard, such asCAN (Controller Area Network), LIN (Local Interconnect Network), LAN(Local Area Network), and FlexRay (registered trademark), a bus, and thelike. The components of the vehicle control system 100 may be directlyconnected without the involvement of the communication network 121.

The input unit 101 includes an apparatus used by the occupant to inputvarious data, instructions, and the like. For example, the input unit101 includes an operation device, such as a touch panel, a button, amicrophone, a switch, and a lever, an operation device that allows inputusing a method other than manual operation, such as sound and gesture,and the like.

In addition, for example, the input unit 101 may be a remote controlapparatus using infrared rays or other radio waves or may be an externalconnection device, such as a mobile device and a wearable device,corresponding to operation of the vehicle control system 100. The inputunit 101 generates an input signal on the basis of data, an instruction,or the like input by the occupant and supplies the input signal to eachcomponent of the vehicle control system 100.

The data acquisition unit 102 includes various sensors and the like thatacquire data to be used for the process of the vehicle control system100 and supplies the acquired data to each component of the vehiclecontrol system 100.

For example, the data acquisition unit 102 includes various sensors fordetecting the state and the like of the user's car. Specifically, thedata acquisition unit 102 includes, for example, a gyrosensor, anacceleration sensor, an inertial measurement apparatus (IMU), andsensors for detecting an amount of operation of an accelerator pedal, anamount of operation of a brake pedal, a steering angle of a steeringwheel, an engine speed, a motor speed, a rotational speed of a wheel,and the like.

In addition, the data acquisition unit 102 includes, for example,various sensors for detecting information outside the user's car.Specifically, the data acquisition unit 102 includes, for example, animaging apparatus, such as a ToF (Time Of Flight) camera, a stereocamera, a monocular camera, an infrared camera, and other cameras. Inaddition, the data acquisition unit 102 includes, for example,environment sensors for detecting weather, climate conditions, and thelike and surrounding information detection sensors for detecting objectsaround the user's car. The environment sensors include, for example, arain sensor, a fog sensor, a sunlight sensor, a snow sensor, and thelike. The surrounding information detection sensors include, forexample, an ultrasonic sensor, a radar, a LiDAR (Light Detection andRanging, Laser Imaging Detection and Ranging), a sonar, and the like.

Furthermore, the data acquisition unit 102 includes, for example,various sensors for detecting a current position of the user's car.Specifically, the data acquisition unit 102 includes, for example, aGNSS receiver or the like that receives a signal from a GNSS (GlobalNavigation Satellite System) satellite.

In addition, the data acquisition unit 102 includes, for example,various sensors for detecting information inside the vehicle.Specifically, the data acquisition unit 102 includes, for example, animaging apparatus that images the driver, a biosensor that detectsbiological information of the driver, a microphone that collects soundof the vehicle interior, and the like. The biosensor is provided on, forexample, the seat surface, the steering wheel, or the like, and thebiosensor detects the biological information of the occupant sitting inthe seat or the driver holding the steering wheel.

The camera included in the data acquisition unit 102 images the scene inthe traveling direction of the vehicle. A scene image taken by thecamera is analyzed to specify the presence or absence of buildings, thebrightness of the buildings, the brightness of the surroundings, and thelike.

In addition, the state of the vehicle, such as the traveling directionand the velocity, is specified on the basis of the results detected byvarious sensors included in the data acquisition unit 102. The specifiedstate of the vehicle is used for prediction or the like of the travelingroute of the vehicle.

The communication unit 103 communicates with the vehicle interior device104 as well as various devices, servers, base stations, and the likeoutside the vehicle to transmit data supplied from each of thecomponents of the vehicle control system 100 and to supply received datato each of the components of the vehicle control system 100. Note thatthe communication protocol supported by the communication unit 103 isnot particularly limited, and the communication unit 103 can alsosupport a plurality of types of communication protocols.

For example, the communication unit 103 uses a wireless LAN, Bluetooth(registered trademark), NFC (Near Field Communication), WUSB (WirelessUSB), or the like to wirelessly communicate with the vehicle interiordevice 104. In addition, the communication unit 103 uses a USB(Universal Serial Bus), HDMI (registered trademark) (High-DefinitionMultimedia Interface), MHL (Mobile High-definition Link), or the like toperform wired communication with the vehicle interior device 104 througha connection terminal (and a cable if necessary) not illustrated.

The communication unit 103 communicates with a device (for example, anapplication server or a control server) existing on an external network(for example, the Internet, a cloud network, or a network specific to aservice provider) through a base station or an access point.

The content to be superimposed on the scene may be acquired from aserver that manages the content. In this case, the communication unit103 communicates with the server to acquire the content. The contentacquired by the communication unit 103 is supplied and stored in, forexample, the storage unit 111.

The communication unit 103 uses a P2P (Peer To Peer) technique tocommunicate with a terminal (for example, a terminal of a pedestrian ora shop, or an MTC (Machine Type Communication) terminal) existing nearthe user's car. In addition, the communication unit 103 performs V2Xcommunication, such as vehicle-to-vehicle communication,vehicle-to-infrastructure communication, vehicle-to-home communication,and vehicle-to-pedestrian communication. The communication unit 103includes a beacon reception unit and receives a radio wave or anelectromagnetic wave transmitted from a wireless station or the likeinstalled on the road to acquire information regarding a currentposition, a traffic jam, a traffic regulation, a required time, and thelike.

The vehicle interior device 104 includes, for example, a mobile deviceor a wearable device possessed by the occupant, an information devicecarried in or attached to the user's car, a navigation apparatus thatsearches for a route to an arbitrary destination, and the like.

The output control unit 105 controls output of various types ofinformation to the occupant of the user's car or to the outside of thevehicle. For example, the output control unit 105 generates an outputsignal including at least one of visual information (for example, imagedata) and auditory information (for example, audio data) and suppliesthe output signal to the output unit 106 to control the output of thevisual information and the auditory information from the output unit106.

Specifically, for example, the output control unit 105 combines imagedata imaged by different imaging apparatuses of the data acquisitionunit 102 to generate a bird's-eye image, a panoramic image, or the likeand supplies an output signal including the generated image to theoutput unit 106. In addition, for example, the output control unit 105generates audio data including warning sound, a warning message, or thelike for danger, such as collision, contact, and entry into a dangerzone, and supplies an output signal including the generated audio datato the output unit 106.

The output unit 106 includes an apparatus that can output the visualinformation or the auditory information to the occupant of the user'scar or to the outside the vehicle. For example, the output unit 106includes a display apparatus, an instrument panel, an audio speaker,headphones, a wearable device such as a head-mounted display worn by theoccupant, a lamp, and the like.

In addition, the output unit 106 includes a projection unit 106A. Theprojection unit 106A is a display device, such as an HUD and atransmissive display, with an AR display function. The projection unit106A projects various types of information, such as content, to thewindshield G as described above.

The drive control unit 107 supplies various control signals to the drivesystem 108 to control the drive system 108. In addition, the drivecontrol unit 107 supplies control signals to components other than thedrive system 108 as necessary to send a notification or the like of thecontrol state of the drive system 108.

The drive system 108 includes various apparatuses regarding the drivesystem of the user's car. For example, the drive system 108 includes adriving force generation apparatus, such as an internal combustionengine and a drive motor, for generating driving force, a driving forcetransmission mechanism for transmitting the driving force to the wheel,a steering mechanism that adjusts the steering angle, a brakingapparatus that generates braking force, an ABS (Antilock Brake System),an ESC (Electronic Stability Control), an electric power steeringapparatus, and the like.

The body control unit 109 generates various control signals and suppliesthe control signals to the body system 110 to control the body system110. In addition, the body control unit 109 supplies the control signalsto components other than the body system 110 as necessary to send anotification or the like of the control state of the body system 110.

The body system 110 includes various apparatuses of the body equipped onthe vehicle body. For example, the body system 110 includes a keylessentry system, a smart key system, a power window apparatus, a powerseat, a steering wheel, an air conditioner, various lamps (for example,a headlamp, a back lamp, a brake lamp, a turn signal, a fog lamp, andthe like), and the like.

The storage unit 111 includes a storage device, such as an SSD (SolidState Drive) and an HDD (Hard Disc Drive). The storage unit 111 storesprograms, data, and the like used by the components of the vehiclecontrol system 100.

For example, map data, such as a three-dimensional highly accurate maplike a dynamic map, a global map covering a wide area with loweraccuracy than the highly accurate map, and a local map includinginformation regarding the surroundings of the user's car, is stored inthe storage unit 111.

The automatic drive control unit 112 performs control regardingautomatic drive, such as autonomous traveling and drive support. Forexample, the automatic drive control unit 112 performs cooperativecontrol to realize functions of ADAS (Advanced Driver Assistance System)including collision avoidance or shock mitigation of the user's car,follow-up traveling on the basis of the following distance, traveling ata constant speed, collision warning of the user's car, lane departurewarning of the user's car, and the like. In addition, the automaticdrive control unit 112 performs cooperative control aimed at automaticdrive or the like for autonomous traveling regardless of the operationof the driver.

The automatic drive control unit 112 includes a detection unit 131, aself-position estimation unit 132, a situation analysis unit 133, aplanning unit 134, and a motion control unit 135.

The detection unit 131 detects various types of information necessaryfor controlling the automatic drive. The detection unit 131 includes avehicle exterior information detection unit 141, a vehicle interiorinformation detection unit 142, and a vehicle state detection unit 143.

The vehicle exterior information detection unit 141 executes a detectionprocess of information outside the user's car on the basis of data or asignal from each component of the vehicle control system 100. Forexample, the vehicle exterior information detection unit 141 executes adetection process, a recognition process, and a tracking process of anobject around the user's car as well as a detection process of thedistance to the object. Examples of the object to be detected include avehicle, a person, an obstacle, a structure, a road, a traffic light, atraffic sign, a road sign, and the like.

In addition, the vehicle exterior information detection unit 141executes a detection process of the environment around the user's car.Examples of the surrounding environment to be detected include weather,temperature, humidity, brightness, a state of a road surface, and thelike. The vehicle exterior information detection unit 141 supplies dataindicating the results of the detection process to the self-positionestimation unit 132, a map analysis unit 151, a traffic rule recognitionunit 152, and a situation recognition unit 153 of the situation analysisunit 133, an emergency avoidance unit 171 of the motion control unit135, and the like.

The vehicle interior information detection unit 142 executes a detectionprocess of information inside the vehicle on the basis of data orsignals from the components of the vehicle control system 100. Forexample, the vehicle interior information detection unit 142 executes anauthentication process and a recognition process of the driver, adetection process of the state of the driver, a detection process of theoccupant, a detection process of the environment inside the vehicle, andthe like. Examples of the state of the driver to be detected includephysical conditions, alertness, concentration, fatigue, a visual linedirection, and the like. Examples of the environment inside the vehicleto be detected include the temperature, humidity, brightness, odor, andthe like. The vehicle interior information detection unit 142 suppliesdata indicating the results of the detection process to the situationrecognition unit 153 of the situation analysis unit 133, the emergencyavoidance unit 171 of the motion control unit 135, and the like.

The vehicle state detection unit 143 executes a detection process of thestate of the user's car on the basis of data or signals from thecomponents of the vehicle control system 100. Examples of the state ofthe user's car to be detected include the velocity, the acceleration,the steering angle, the presence/absence and details of abnormality, thestate of driving operation, the position and inclination of a powerseat, the state of a door lock, the state of other on-board devices, andthe like. The vehicle state detection unit 143 supplies data indicatingthe results of the detection process to the situation recognition unit153 of the situation analysis unit 133, the emergency avoidance unit 171of the motion control unit 135, and the like.

The self-position estimation unit 132 executes an estimation process ofa position, an orientation, and the like of the user's car on the basisof data or signals from the components of the vehicle control system100, such as the vehicle exterior information detection unit 141 and thesituation recognition unit 153 of the situation analysis unit 133.

In addition, the self-position estimation unit 132 generates a local mapused for estimating the self-position (hereinafter, referred to as aself-position estimation map) as necessary. The self-position estimationmap is a highly accurate map using, for example, a technique such asSLAM (Simultaneous Localization and Mapping).

The self-position estimation unit 132 supplies data indicating theresults of the estimation process to the map analysis unit 151, thetraffic rule recognition unit 152, and the situation recognition unit153 of the situation analysis unit 133 and the like. In addition, theself-position estimation unit 132 causes the storage unit 111 to storethe self-position estimation map.

The situation analysis unit 133 executes an analysis process of thesituation of the user's car and the surroundings. The situation analysisunit 133 includes the map analysis unit 151, the traffic rulerecognition unit 152, the situation recognition unit 153, and asituation prediction unit 154.

The map analysis unit 151 executes an analysis process of various mapsstored in the storage unit 111 while using data or signals from thecomponents of the vehicle control system 100, such as the self-positionestimation unit 132 and the vehicle exterior information detection unit141, as necessary and constructs a map including information necessaryfor the process of automatic drive. The map analysis unit 151 suppliesthe constructed map to, for example, the traffic rule recognition unit152, the situation recognition unit 153, and the situation predictionunit 154 as well as a route planning unit 161, an action planning unit162, and a motion planning unit 163 of the planning unit 134.

The traffic rule recognition unit 152 executes a recognition process oftraffic rules around the user's car on the basis of data or signals fromthe components of the vehicle control system 100, such as theself-position estimation unit 132, the vehicle exterior informationdetection unit 141, and the map analysis unit 151. As a result of therecognition process, for example, the position and the state of signalsaround the user's car, the details of traffic regulations around theuser's car, driving lanes that can be traveled, and the like arerecognized. The traffic rule recognition unit 152 supplies dataindicating the results of the recognition process to the situationprediction unit 154 and the like.

The situation recognition unit 153 executes a recognition process of thesituation regarding the user's car on the basis of data or signals fromthe components of the vehicle control system 100, such as theself-position estimation unit 132, the vehicle exterior informationdetection unit 141, the vehicle interior information detection unit 142,the vehicle state detection unit 143, and the map analysis unit 151. Forexample, the situation recognition unit 153 executes a recognitionprocess of the situation of the user's car, the situation around theuser's car, the situation of the driver of the user's car, and the like.In addition, the situation recognition unit 153 generates a local mapused for recognizing the situation around the user's car (hereinafter,referred to as a situation recognition map) as necessary. The situationrecognition map is, for example, an occupancy grid map.

Examples of the situations of the user's car to be recognized includethe position, the orientation, and the motion (for example, a velocity,an acceleration, a movement direction, and the like) of the user's caras well as the presence/absence and the details of abnormality. Examplesof the situation around the user's car to be recognized include a typeand a position of a surrounding stationary object, a type, a position,and motion (for example, a velocity, an acceleration, a movementdirection, and the like) of a surrounding moving object, a configurationof a surrounding road, the state of the road surface, and the weather,the temperature, the humidity, and the brightness of the surroundings.Examples of the state of the driver to be recognized include physicalconditions, alertness, concentration, fatigue, motion of a visual line,and driving operation.

The situation recognition unit 153 supplies data (including thesituation recognition map as necessary) indicating the results of therecognition process to the self-position estimation unit 132, thesituation prediction unit 154, and the like. In addition, the situationrecognition unit 153 causes the storage unit 111 to store the situationrecognition map.

The situation prediction unit 154 executes a prediction process of thesituation regarding the user's car on the basis of data or signals fromthe components of the vehicle control system 100, such as the mapanalysis unit 151, the traffic rule recognition unit 152, and thesituation recognition unit 153. For example, the situation predictionunit 154 executes a prediction process of the situation of the user'scar, the situation around the user's car, the situation of the driver,and the like.

The situation of the user's car to be predicted includes, for example,behavior of the user's car, generation of abnormality, a possible traveldistance, and the like. The situation around the user's car to bepredicted includes, for example, behavior of a moving object around theuser's car, change in the state of a signal, change in the state of theenvironment such as the weather, and the like. The situation of thedriver to be predicted includes, for example, behavior and physicalconditions of the driver, and the like.

The situation prediction unit 154 supplies data indicating the resultsof the prediction process to, for example, the route planning unit 161,the action planning unit 162, and the motion planning unit 163 of theplanning unit 134 along with the data from the traffic rule recognitionunit 152 and the situation recognition unit 153.

The route planning unit 161 plans a route to the destination on thebasis of data or signals from the components of the vehicle controlsystem 100 such as the map analysis unit 151 and the situationprediction unit 154. For example, the route planning unit 161 sets theroute from the current position to the designated destination on thebasis of the global map. In addition, the route planning unit 161appropriately changes the route on the basis of, for example, thesituation of a traffic jam, an accident, a traffic regulation, aconstruction work, and the like, the physical conditions of the driver,and the like. The route planning unit 161 supplies data indicating theplanned route to the action planning unit 162 and the like.

The action planning unit 162 plans actions of the user's car for safelytraveling the route planned by the route planning unit 161 in a plannedperiod of time on the basis of data or signals from the components ofthe vehicle control system 100 such as the map analysis unit 151 and thesituation prediction unit 154. For example, the action planning unit 162plans start, stop, a traveling direction (for example, forward,backward, left turn, right turn, change of the direction, and the like),a driving lane, a traveling speed, passing, and the like. The actionplanning unit 162 supplies data indicating the planned actions of theuser's car to the motion planning unit 163 and the like.

The motion planning unit 163 plans motions of the user's car forrealizing the actions planned by the action planning unit 162 on thebasis of data or signals from the components of the vehicle controlsystem 100 such as the map analysis unit 151 and the situationprediction unit 154. For example, the motion planning unit 163 plansacceleration, deceleration, a traveling path, and the like. The motionplanning unit 163 supplies data indicating the planned motions of theuser's car to an acceleration and deceleration control unit 172 and adirection control unit 173 of the motion control unit 135, and the like.

The motion control unit 135 controls the motions of the user's car. Themotion control unit 135 includes the emergency avoidance unit 171, theacceleration and deceleration control unit 172, and the directioncontrol unit 173.

The emergency avoidance unit 171 executes a detection process ofemergency, such as collision, contact, entry into a danger zone,abnormality of a driver, and abnormality of a vehicle, on the basis ofthe detection results of the vehicle exterior information detection unit141, the vehicle interior information detection unit 142, and thevehicle state detection unit 143. The emergency avoidance unit 171 plansmotions of the user's car for avoiding emergency, such as sudden stopand sharp turn, in a case where emergency is detected. The emergencyavoidance unit 171 supplies data indicating the planned motions of theuser's car to the acceleration and deceleration control unit 172, thedirection control unit 173, and the like.

The acceleration and deceleration control unit 172 controls theacceleration and the deceleration for realizing the motions of theuser's car planned by the motion planning unit 163 or the emergencyavoidance unit 171. For example, the acceleration and decelerationcontrol unit 172 computes control target values of the driving forcegeneration apparatus or the braking apparatus for realizing the plannedacceleration, deceleration, or sudden stop and supplies a controlcommand indicating the computed control target values to the drivecontrol unit 107.

The direction control unit 173 controls the direction for realizing themotions of the user's car planned by the motion planning unit 163 or theemergency avoidance unit 171. For example, the direction control unit173 computes control target values of the steering mechanism forrealizing the traveling path or sharp turn planned by the motionplanning unit 163 or the emergency avoidance unit 171 and supplies acontrol command indicating the computed control target values to thedrive control unit 107.

<1-2-2. Configuration of Output Control Unit>

FIG. 9 is a block diagram illustrating a functional configurationexample of the output control unit 105 of FIG. 8 . At least part offunctional units illustrated in FIG. 9 is realized by executing apredetermined program.

The output control unit 105 includes an information superimpositionappropriate visual field setting unit 201, a superimposition targetframe selection unit 202, a display control unit 203, and a contentacquisition unit 204.

The information superimposition appropriate visual field setting unit201 analyzes an image obtained by imaging the scene in the travelingdirection of the vehicle. The information superimposition appropriatevisual field setting unit 201 is supplied with, for example, a sceneimage taken by the camera included in the data acquisition unit 102.

The information superimposition appropriate visual field setting unit201 sets information superimposition possible frames on the wallsurfaces of buildings in the scene image. The informationsuperimposition possible frame is a region of the surface (regioncorresponding to the surface) of an object, such as a building, in whichthe content can be superimposed.

In addition, the information superimposition appropriate visual fieldsetting unit 201 selects predetermined frames of the informationsuperimposition possible frames as information superimpositionappropriate frames that are frames suitable for superimposing thecontent.

That is, the content is not superimposed on all of the informationsuperimposition possible frames, and the information superimpositionpossible frames suitable for superimposing the content are selectedaccording to the traveling state and the like. Information regarding theinformation superimposition appropriate frames set by the informationsuperimposition appropriate visual field setting unit 201 is supplied tothe superimposition target frame selection unit 202. Information of theanalysis results of the scene image is appropriately supplied from theinformation superimposition appropriate visual field setting unit 201 tothe display control unit 203.

The superimposition target frame selection unit 202 selects asuperimposition location of the content acquired by the contentacquisition unit 204 from the information superimposition appropriateframes set by the information superimposition appropriate visual fieldsetting unit 201. Information regarding the information superimpositionappropriate frame selected as the superimposition location of thecontent is supplied to the display control unit 203.

The display control unit 203 applies image processing to the contentaccording to the situation of the information superimpositionappropriate frame selected as the superimposition location to adjust theappearance. In this way, the display control unit 203 has a function ofgenerating visual information for displaying the content. In addition,the display control unit 203 controls the projection unit 106A tosuperimpose, on the information superimposition appropriate frame, thecontent subjected to the image processing. The display control unit 203projects the content to the region of the windshield G corresponding tothe information superimposition appropriate frame to display thecontent.

The content acquisition unit 204 reads the content from the storage unit111 to acquire the content. For example, content acquired from a servernot illustrated is stored in the storage unit 111. The content acquiredby the content acquisition unit 204 is supplied to the superimpositiontarget frame selection unit 202 and the display control unit 203.

In this way, the content is displayed by executing the process of theinformation superimposition appropriate visual field setting unit 201 asa process of a first stage, executing the process of the superimpositiontarget frame selection unit 202 as a process of a second stage, andexecuting the process of the display control unit 203 as a process of athird stage.

Hereinafter, details of the processes of the stages will be describedalong with configurations of components of the informationsuperimposition appropriate visual field setting unit 201, thesuperimposition target frame selection unit 202, and the display controlunit 203.

<1-2-3. Process of First Stage (Setting of Information SuperimpositionAppropriate Visual Field)>

FIG. 10 is a block diagram illustrating a configuration example of theinformation superimposition appropriate visual field setting unit 201 ofFIG. 9 .

The information superimposition appropriate visual field setting unit201 includes an image analysis unit 211, a light state mode setting unit212, an object detection unit 213, a frame setting unit 214, and anappropriate visual field setting unit 215.

The image analysis unit 211 analyzes a scene image obtained by imaging.As a result of the analysis of the scene image, the outlines of thebuildings are detected, and the brightness (luminance) of thesurroundings is detected. In addition, as a result of the analysis ofthe scene image, the state of sunlight, the state of lighting, the stateof atmosphere, and the like are also detected. Information indicatingthe analysis results of the image analysis unit 211 is supplied to thelight state mode setting unit 212 and the object detection unit 213 andis also supplied to the display control unit 203.

The light state mode setting unit 212 sets a light state mode on thebasis of information supplied from the image analysis unit 211. Forexample, one of a “daytime mode,” a “dusk mode,” and a “night mode” isset as the light state mode.

For example, the “daytime mode” is set when the sun is in the sky sothat the surroundings are bright. The “dusk mode” is set when thesurroundings are a little dark, such as in the evening and at dawn. The“night mode” is set when the sun is set so that the surroundings aredark.

A reference threshold of brightness is set for each mode. The lightstate mode setting unit 212 compares the brightness of the surroundingsspecified by analyzing the image and the brightness as the threshold toset the light state mode according to the current situation ofbrightness. Information regarding the light state mode set by the lightstate mode setting unit 212 is supplied to the frame setting unit 214.

The light state mode setting unit 212 may set the light state mode onthe basis of the detection results of the sensors included in the dataacquisition unit 102, instead of the analysis results of the sceneimage. In addition, the light state mode may be set with reference tothe current time or the like.

The object detection unit 213 acquires the map data and plots thebuildings with detected outlines on the map to thereby generate athree-dimensional model of the buildings which are along the drivingroad and are included in the visual field of the user.

For example, the driving road is specified on the basis of positionmeasurement results of the GNSS receiver included in the dataacquisition unit 102. The map data supplied to the object detection unit213 may be stored in the storage unit 111 or may be acquired from aserver not illustrated.

FIG. 11 is a diagram illustrating an example of the three-dimensionalmodel of the objects along the road.

In the example of FIG. 11 , buildings are lined up on the left and rightof a linear road traveled by the user's vehicle. Such athree-dimensional model including the data of the outlines of thebuildings is generated by the object detection unit 213. Thethree-dimensional model also includes data of outlines of objects otherthan the buildings included in the scene image, such as trees andtelephone poles.

In a case where the map data supplied to the object detection unit 213includes not only information regarding the position of each building,but also information regarding the height and the like of the building,the three-dimensional model as illustrated in FIG. 11 may be generatedon the basis of the map data. In this case, the object detection unit213 determines whether or not the buildings on the map data actuallyexist on the basis of the analysis results of the scene image of theimage analysis unit 211 and identifies the positions of the buildingsthat actually exist.

For example, a building existing on the map data may be destroyed. In acase where the same building as the building on the map data is not inthe scene image, the object detection unit 213 excludes the buildingfrom the target of superimposition of the content.

In this way, whether the objects exist can be checked on the basis ofthe scene image, thereby preventing superimposition of the content on abuilding not in the visual field of the user because the building is notactually there.

In general, the map data is updated every predetermined period, such asevery year, and a building existing on the map data may not be there atthe time of the traveling of the vehicle. Although the content looksfloating in a case where the content is superimposed on the wall surfaceof the building that is not there at the time of the traveling of thevehicle, such an unnatural appearance can be prevented.

The object detection unit 213 outputs information of thethree-dimensional model of the buildings actually in the scene image tothe frame setting unit 214.

The frame setting unit 214 sets the information superimposition possibleframes on the wall surfaces of the buildings included in the visualfield of the user on the basis of the three-dimensional model suppliedfrom the object detection unit 213. For example, planes with areas widerthan a threshold are set as the information superimposition possibleframes.

FIG. 12 is a diagram illustrating an example of setting the informationsuperimposition possible frames.

In the example of FIG. 12 , an information superimposition possibleframe F_01 is set for a building on the near side on the right of acurve. In addition, there are three buildings on the left side along thecurve, and information superimposition possible frames F_02, F_03, andF_04 are set from the buildings on the near side. Note that the sceneillustrated in FIG. 12 is a scene at a vehicle position A describedlater.

In this way, the information superimposition possible frames ascandidates for the superimposition locations of the content are set onthe basis of the three-dimensional model of the buildings viewed fromthe road traveled by the user's vehicle. The frame as thesuperimposition location of the content is selected from the informationsuperimposition possible frames set by the frame setting unit 214.Although one information superimposition possible frame is set on onesurface of the building in FIG. 12 , a plurality of informationsuperimposition possible frames may be set.

In addition, the frame setting unit 214 excludes the informationsuperimposition possible frames not appropriate for the superimpositionlocations of the content among the information superimposition possibleframes set in this way. The information superimposition possible framesare excluded on the basis of the light state mode set by the light statemode setting unit 212, the brightness of each part of the building inthe scene image analyzed by the image analysis unit 211, and the like.

<Determination on Basis of Exclusion Rules> Exclusion Rule 1

The frame setting unit 214 excludes parts in an intense light emittingstate, such as a large monitor, from the information superimpositionpossible frames in all cases of the “daytime mode,” the “dusk mode,” andthe “night mode.” For example, parts of buildings with luminance equalto or greater than a threshold are specified as the parts in the intenselight emitting state.

In this case, whether or not to exclude the part from the informationsuperimposition possible frames is determined on the basis of absoluteluminance of each part.

FIG. 13 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by the exclusion rule 1.

As illustrated in FIG. 13 , large monitors are installed on a part of aregion A1 in an upper section of a surface b11-1 included in a buildingB11 and a part of a region A2 in an upper section of a surface b12-2included in a building B12. In this case, the part of the region A1 andthe part of the region A2 are excluded from the informationsuperimposition possible frames.

In a case where the content is superimposed on a part in the lightemitting state, such as a large monitor, it may seem like the light isleaking from around the content. The part can be excluded from theinformation superimposition possible frames according to the exclusionrule 1 to prevent such an unnatural appearance.

Exclusion Rule 2

The frame setting unit 214 excludes, from the informationsuperimposition possible frames, parts darkly shadowed due to thesunlight in the case of the “daytime mode.” In a case where there are apart with high luminance caused by sunshine and a part in which thedifference in luminance from the luminance of the part is equal to orgreater than a threshold, the part with low luminance is specified asthe part darkly shadowed due to the sunlight.

In this case, whether or not to exclude the part from the informationsuperimposition possible frames is determined on the basis of theabsolute luminance of each part and the difference in luminance from thesurroundings.

FIG. 14 is a diagram illustrating an example of the informationsuperimposition frames excluded by the exclusion rule 2.

As illustrated in FIG. 14 , it is assumed that the sunlight shines onthe surface b11-1 included in the building B11, and an entire surfaceb11-2 that is another surface is shadowed so that the luminancedifference between the surfaces is equal to or greater than thethreshold. In this case, a part of a region A11 of the surface b11-2with low luminance is excluded from the information superimpositionpossible frames.

Similarly, a part of a region A12 of the surface b12-2 included in thebuilding B12 and a part of a region A13 of a surface b13-2 included in abuilding B13 are excluded from the information superimposition possibleframes. The luminance difference is equal to or greater than thethreshold between the upper section of the surface b12-1 and the surfaceb12-2 in the building B12, and the luminance difference is equal to orgreater than the threshold between the upper section of the surfaceb13-1 and the surface b13-2 in the building B13.

The content is hard to see in a case where the content is superimposedon a shadow part where there is a very bright part in the surroundings.The part can be excluded from the information superimposition possibleframes according to the exclusion rule 2 to ensure the visibility of thecontent.

Exclusion Rule 3

The frame setting unit 214 excludes parts that look particularly darkdue to various reasons from the information superimposition possibleframes in the case of the “dusk mode.” For example, parts with luminancelower than a threshold are specified as the parts that look particularlydark.

In this case, whether or not to exclude the parts from the informationsuperimposition possible frames is determined on the basis of theabsolute luminance of each part as in the exclusion rule 1.

FIG. 15 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by the exclusion rule 3.

As illustrated in FIG. 15 , it is assumed that the luminance of asurface b21-1 included in a building B21 is lower than the threshold. Inaddition, it is assumed that a surface b22-1 and a surface b22-2included in a building B22 are shadowed by the building B21, and theluminance is lower than the threshold. In this case, parts of regionsA21-1 and A21-2 of the surface b21-1, a part of a region A22-1 of thesurface b22-1, and a part of a region A22-2 of the surface b22-2 areexcluded from the information superimposition possible frames.

In a case where the content is superimposed on the particularly darkpart, the content is hard to see. The visibility of the content can alsobe ensured by excluding the information superimposition possible framesaccording to the exclusion rule 3.

Note that, although a surface that can be viewed on the front side of abuilding B23 is a little dark in FIG. 15 , the surface is not a partthat looks particularly dark. Therefore, the surface is not excluded.

Exclusion Rule 4

The frame setting unit 214 excludes parts darkly shadowed byillumination light from the information superimposition possible framesin the case of the “night mode.” For example, in a case where there area part with high luminance due to illumination light and a part in whichthe difference in luminance from the luminance of the part is equal toor greater than a threshold, the part with low luminance is specified asthe part darkly shadowed by the illumination light.

In this case, whether or not to remove the part from the informationsuperimposition possible frames is determined on the basis of theabsolute luminance of each part and the difference in luminance from thesurroundings as in the exclusion rule 2.

FIG. 16 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by the exclusion rule 4.

As illustrated in FIG. 16 , it is assumed that light from a street lampL shines on a surface b31-1 included in a building B31, and an entiresurface b31-2 that is another surface is shadowed so that the luminancedifference between the surfaces is equal to or greater than a threshold.In this case, a part of a region A31 of the surface b31-2 with lowluminance is excluded from the information superimposition possibleframes.

Similarly, it is assumed that the light from the street lamp L shines onpart of a surface b32-1 included in a building B32, and another part isshadowed by the building B31 so that the luminance difference betweenthe parts is equal to or greater than the threshold. In this case, apart of a region A32 with low luminance in the surface 32-1 is excludedfrom the information superimposition possible frames.

The content is hard to see in a case where the content is superimposedon a shadow part where there is a very bright part in the surroundings.The visibility of the content can also be ensured by excluding theinformation superimposition possible frames according to the exclusionrule 4.

Exclusion Rule 5

The frame setting unit 214 excludes parts in the light emitting state inouter walls of buildings and window parts in the light emitting statedue to internal lighting or the like from the informationsuperimposition possible frames in the case of the “night mode.” Forexample, parts of buildings with luminance equal to or greater than athreshold are specified as the parts in the light emitting state.

In this case, whether or not to exclude the parts from the informationsuperimposition possible frames is determined on the basis of theabsolute luminance of each part.

FIG. 17 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by the exclusion rule 5.

As illustrated in FIG. 17 , it is assumed that parts of regions A41-1and A41-2 of the surface b11-1 included in the building B11 and parts ofregions A41-3 and A41-4 of the surface b11-2 include windows, and theparts are in the light emitting state due to lighting in the room. Inthis case, the parts of the regions A41-1 to A41-4 are excluded from theinformation superimposition possible frames.

Similarly, a part of a region A42 of the surface b12-2 included in thebuilding B12 and a part of a region A43 of the surface b13-2 included inthe building B13 are excluded from the information superimpositionpossible frames. It is assumed that the parts also include windows, andthe parts are in the light emitting state due to lighting in the room.

In a case where the surroundings are dark, it may seem like the light isleaking from around the content in a case where the content issuperimposed on a window part or the like where the light inside theroom is leaking. The information superimposition possible frames can beexcluded according to the exclusion rule 5 to prevent such an unnaturalappearance.

Exclusion Rule 6

The frame setting unit 214 excludes parts morally or customarilyinappropriate for superimposing the content, such as a historicalstructure, a display object like a signboard, and an object at asightseeing spot, from the information superimposition possible frames.The exclusion rule 6 is adopted in all of the cases of the “daytimemode,” the “dusk mode,” and the “night mode.”

FIG. 18 is a diagram illustrating an example of the informationsuperimposition possible frames excluded by the exclusion rule 6.

As indicated by regions A51 and A52 of FIG. 18 , for example, outerwalls of a castle as a historical structure are excluded from theinformation superimposition possible frames.

The determination on the basis of the exclusion rule 6 is performed bythe frame setting unit 214 on the basis of, for example, the informationincluded in the map data. The scene image may be analyzed to detect ahistorical structure and the like to determine whether or not to removethe regions from the information superimposition possible frames.

The frame setting unit 214 excludes the information superimpositionpossible frames according to the rules described above and outputsinformation regarding the remaining (not excluded) informationsuperimposition possible frames to the appropriate visual field settingunit 215.

<Determination of Information Superimposition Appropriate Frames>

The appropriate visual field setting unit 215 of FIG. 10 determineswhether or not the information superimposition possible frames set bythe frame setting unit 214 and determined on the basis of the exclusionrules satisfy the following conditions 1 to 4.

Condition 1: The distance from the user, the angle with respect to thevehicle traveling direction, and the angular velocity in a user visualfield calculated from the vehicle speed are within reference ranges.

Condition 2: The angle with respect to the vehicle traveling directionis equal to or greater than a minimum reference value.

Condition 3: The exposed area ratio is equal to or greater than areference value in a case where the information superimposition possibleframe can be viewed beyond a front object, such as a tree and atelephone pole.

Condition 4: The stay time in the user visual field is equal to orgreater than a reference value.

Whether or not these conditions are satisfied is determined byappropriately using the analysis results of the scene image, theinformation regarding the road included in the map data, the travelingdirection of the vehicle, the vehicle speed, and the like. Informationregarding the current state of the vehicle, such as the travelingdirection of the vehicle and the vehicle speed, is specified on thebasis of, for example, information supplied from the situation analysisunit 133 and the like. In addition, information regarding the state ofthe vehicle in the future is specified on the basis of, for example,progress state prediction information supplied from the motion planningunit 163 and the like.

The appropriate visual field setting unit 215 sets the informationsuperimposition possible frame satisfying all of the conditions 1 to 4as an information superimposition appropriate frame. The informationsuperimposition possible frame not satisfying at least any one of theconditions 1 to 4 is not selected as an information superimpositionappropriate frame.

Although the determination of whether or not the conditions 1 to 4 aresatisfied will be described, a specific example of a prerequisite stateof the vehicle will be described first.

FIG. 19 is a diagram illustrating an example of transition of thevehicle position.

FIG. 19 illustrates a state of transition in the position of the vehicletraveling a curve to the right, as viewed from a plane. A rectangle withrounded corners on the road represents the vehicle, and a rectangle onboth sides of the road represents a building. There is one buildinginside the curve, and the information superimposition possible frameF_01 is set on the surface on the road side of the building. Inaddition, there are three buildings outside the curve, and theinformation superimposition possible frames F_02, F_03, and F_04 are seton the buildings from the near side. In FIG. 19 , a thick line in therectangle representing the building indicates the informationsuperimposition possible frame.

Vehicle positions A to E represent the positions of the vehicle at eachtime. The vehicle sequentially travels on a trajectory connecting thevehicle positions A to E.

For example, based on the orientation of the vehicle at the vehicleposition A, the direction change at the vehicle position B is expressedas 22.5°, and the direction change at the vehicle position C isexpressed as 45° as illustrated in FIG. 20 . The direction change of thevehicle at the vehicle position D is expressed as 67.5°, and thedirection change of the vehicle at the vehicle position E is expressedas 90°.

An interior angle of a sector indicated on the basis of the vehicleposition represents a range of the visual field of the user at eachposition of the vehicle. For example, the visual field at the vehicleposition A is indicated by a sector #1, and the visual field at thevehicle position B is indicated by a sector #2. The visual field at thevehicle position C is indicated by a sector #3. In this way, the sceneviewed by the user changes according to the vehicle position and thedirection of the vehicle.

The state of the vehicle at each time is generated at each timing on thebasis of the current state of progress of the vehicle and the state ofprogress in the future. FIG. 19 illustrates the state of transition inthe vehicle position generated at a timing that the vehicle is travelingat the vehicle position A. The states of the vehicle at the vehiclepositions B to E are states predicted on the basis of the state ofprogress in the future. The state of progress in the future is indicatedby the progress state prediction information input from the outside.

FIG. 21 depicts diagrams illustrating scene images at the vehiclepositions.

For the convenience of description, it is assumed that the range of thescene image captured by the camera matches the range of the visual fieldof the user. The entire scene viewed by the user is imaged by thecamera.

A of FIG. 21 illustrates a scene image at the vehicle position A. Thescene image at the vehicle position A includes all of the informationsuperimposition possible frames F_01 to F_04. As indicated by the sector#1 of FIG. 19 , the visual field of the user at the timing of travelingat the vehicle position A includes the information superimpositionpossible frames F_01 to F_04.

In the state of A in FIG. 21 , respective trees are planted at theposition in front of the building provided with the informationsuperimposition possible frame F_03 and at the position in front of thebuilding provided with the information superimposition possible frameF_04 in a manner that the trees partially overlap the informationsuperimposition possible frames. Note that the scene illustrated in A ofFIG. 21 is the same as the scene described with reference to FIG. 12 .

B of FIG. 21 illustrates a scene image at the vehicle position B. Thecurrent vehicle position is the vehicle position A, and therefore, thescene image illustrated in B of FIG. 21 represents a scene predicted tobe viewed after a lapse of certain time.

The scene image at the vehicle position B includes the informationsuperimposition possible frame F_03 and the information superimpositionpossible frame F_04. As indicated by the sector #2 in FIG. 19 , thevisual field of the user at the timing of traveling at the vehicleposition B includes the information superimposition possible frame F_03and the information superimposition possible frame F_04.

C of FIG. 21 illustrates a scene image at the vehicle position C. Thecurrent vehicle position is the vehicle position A, and therefore, thescene image illustrated in C of FIG. 21 represents a scene predicted tobe viewed after an additional lapse of certain time from the timing oftraveling at the vehicle position B.

The scene image at the vehicle position C includes the informationsuperimposition possible frame F_04. As indicated by the sector #3 inFIG. 19 , the information superimposition possible frame F_04 isincluded on the left side of the visual field of the user at the timingof traveling at the vehicle position C.

At the timing of traveling at the vehicle positions D and E, all of theinformation superimposition possible frames F_01 to F_04 are out of thevisual field of the user.

Determination of Condition 1

The appropriate visual field setting unit 215 determines that thecondition 1 is satisfied in a case where the distance from the user, theangle with respect to the vehicle traveling direction, and the angularvelocity in the user visual field calculated from the vehicle speed arewithin reference ranges. The angular velocity in the user visual fieldrepresents an angular velocity of the vehicle during the time that theinformation superimposition possible frame is within the visual field ofthe user on the basis of the position of each informationsuperimposition possible frame.

FIG. 22 is a diagram illustrating an example of the angular velocity inthe user visual field.

The example of FIG. 22 illustrates determination using the angularvelocity in the user visual field during movement from the vehicleposition A to the vehicle position B. In this case, as illustrated inFIG. 22 , the angular velocity in the user visual field of theinformation superimposition possible frame F_01 is 41°/sec, and theangular velocity in the user visual field of the informationsuperimposition possible frame F_02 is 38°/sec. In addition, the angularvelocity in the user visual field in the information superimpositionpossible frame F_03 is 3°/sec, and the angular velocity in the uservisual field of the information superimposition possible frame F_04 is8°/sec.

For example, in a case where an upper limit of the reference value ofthe angular velocity in the user visual field is 30°/sec, theinformation superimposition possible frame F_01 and the informationsuperimposition possible frame F_02 exceeding the reference value do notsatisfy the condition 1, and it is determined that the frames areinappropriate for the frames to be provided with the content.

In contrast, the information superimposition possible frame F_03 and theinformation superimposition possible frame F_04 within the range of thereference value satisfy the condition 1, and it is determined that theframes are appropriate for the frames to be provided with the content.The fact that the angular velocity in the user visual field is smallerthan the reference value indicates that the amount of movement in thevisual field of the user is small.

In this way, whether or not the frame is appropriate for the frame to beprovided with the content can be determined on the basis of thecondition 1 to select a frame with a small amount of movement in thevisual field of the user, that is, a frame that can be easily viewedwhen the content is superimposed.

For example, in a case where the content is superimposed on theinformation superimposition possible frame with a large angular velocityin the user visual field, the movement of the content looks large. Thedetermination on the basis of the condition 1 can prevent thesuperimposition of the content on the information superimpositionpossible frame that moves large in the visual field.

Determination of Condition 2

The appropriate visual field setting unit 215 determines that thecondition 2 is satisfied in a case where the angle with respect to thevehicle traveling direction is equal to or greater than a minimumreference value.

FIG. 23 is a diagram illustrating an example of the angle with respectto the vehicle traveling direction.

As illustrated in FIG. 23 , the setting surface of the informationsuperimposition possible frame F_03 is indicated by a broken line L1,and the setting surface of the information superimposition possibleframe F_04 is indicated by a broken line L2. The informationsuperimposition possible frame F_01 and the information superimpositionpossible frame F_02 are determined as inappropriate frames in thedetermination on the basis of the condition 1, and therefore, thedetermination on the basis of the condition 2 is not performed for theinformation superimposition possible frame F_01 and the informationsuperimposition possible frame F_02.

The angle of the information superimposition possible frame F_03 withrespect to the vehicle traveling direction at the vehicle position A isindicated by a sector #11-1, and the angle of the informationsuperimposition possible frame F_04 with respect to the vehicletraveling direction is indicated by a sector #11-2. The angles of theinformation superimposition possible frames with respect to the vehicletraveling direction are both 90°.

In the example, the information superimposition possible frame F_03 andthe information superimposition possible frame F_04 are set on parallelsurfaces, and the angles with respect to the vehicle traveling directionare the same angle at the same vehicle position. In a case where theframes are set on surfaces that are not parallel, the angles withrespect to the vehicle traveling direction are different angles.

The angles of the information superimposition possible frames withrespect to the vehicle traveling direction at each vehicle position aresimilarly obtained.

That is, the angle of the information superimposition possible frameF_03 with respect to the vehicle traveling direction at the vehicleposition B is indicated by a sector #12-1, and the angle of theinformation superimposition possible frame F_04 with respect to thevehicle traveling direction is indicated by a sector #12-2. The anglesof the information superimposition possible frames with respect to thevehicle traveling direction are both 67.5°.

The angle of the information superimposition possible frame F_04 withrespect to the vehicle traveling direction at the vehicle position C isindicated by a sector #13. The angle of the information superimpositionpossible frame F_04 with respect to the vehicle traveling direction is45°. The determination using the angle of the informationsuperimposition possible frame F_03 with respect to the vehicletraveling direction is not performed because the frame is already out ofthe visual field of the user at the timing of traveling at the vehicleposition C.

For example, in a case where the reference value of the minimum anglewith respect to the vehicle traveling direction is 30°, the angles ofthe information superimposition possible frame F_03 and the informationsuperimposition possible frame F_04 with respect to the vehicletraveling direction both exceed the reference value at the timing oftraveling at the vehicle position A. The information superimpositionpossible frame F_03 and the information superimposition possible frameF_04 satisfy the condition 2, and it is determined that the frames areappropriate for the frames to be provided with the content.

In addition, both of the angles of the information superimpositionpossible frame F_03 and the information superimposition possible frameF_04 with respect to the vehicle traveling direction also exceed thereference value at the timing of traveling at the vehicle position B.The information superimposition possible frame F_03 and the informationsuperimposition possible frame F_04 satisfy the condition 2, and it isdetermined that the frames are appropriate for the frames to be providedwith the content.

The angle of the information superimposition possible frame F_04 withrespect to the vehicle traveling direction exceeds the reference valueat the timing of traveling at the vehicle position C. The informationsuperimposition possible frame F_04 satisfies the condition 2, and it isdetermined that the frame is appropriate for the frame to be providedwith the content.

An information superimposition possible frame in which the angle withrespect to the vehicle traveling direction exceeds the reference valuenot only at one timing, but also at a plurality of timings may bedetermined to satisfy the condition 2.

The fact that the angle of the information superimposition possibleframe with respect to the vehicle traveling direction is equal to orgreater than the reference value indicates that the informationsuperimposition possible frame is set in a direction close to the frontof the user.

In this way, whether or not the frame is appropriate for the frame to beprovided with the content can be determined on the basis of thecondition 2 to select a frame set in a direction close to the front ofthe user, that is, a frame that can be easily viewed when the content issuperimposed.

Determination of Condition 3 The appropriate visual field setting unit215 determines that the condition 3 is satisfied in a case where theexposed area ratio is equal to or greater than a reference value whenthe information superimposition possible frame can be viewed beyond afront object, such as a tree and a telephone pole. The front object isan object in front of the information superimposition possible frame asviewed from the position of the user.

Note that the determination on the basis of the condition 3 may beperformed in a case where the distance between the front object and abuilding (information superimposition possible frame) behind the frontobject is closer than a threshold. The distance between the front objectand the building is specified on the basis of, for example, thethree-dimensional model of the object generated by the object detectionunit 213.

FIGS. 24 and 25 are diagrams each illustrating an example of the exposedarea ratio.

As illustrated in FIG. 24 , for example, there is a tree T1 in front ofthe information superimposition possible frame F_03 included in thevisual field of the user, and there is a tree T2 in front of theinformation superimposition possible frame F_04 at the timing oftraveling at the vehicle position A. As illustrated in FIG. 25 , theuser views the information superimposition possible frames partiallycovered by the trees.

The exposed area ratio of the information superimposition possible frameF_03 is indicated as a ratio of the area of the part (diagonal line partin FIG. 25 ) not hidden by the tree T1 to the area of the entireinformation superimposition possible frame F_03. The areas are obtainedby, for example, the appropriate visual field setting unit 215 on thebasis of the analysis results of the scene image. In this example, theexposed area ratio of the information superimposition possible frameF_03 is 80%.

Similarly, the exposed area ratio of the information superimpositionpossible frame F_04 is indicated as a ratio of the area of the part nothidden by the tree T2 to the area of the entire informationsuperimposition possible frame F_04. In this example, the exposed arearatio of the information superimposition possible frame F_04 is 95%.

For example, in a case where the minimum reference value of the exposedarea ratio is 60%, both of the exposed area ratios of the informationsuperimposition possible frame F_03 and the information superimpositionpossible frame F_04 exceed the reference value at the timing oftraveling at the vehicle position A. The information superimpositionpossible frame F_03 and the information superimposition possible frameF_04 satisfy the condition 3, and it is determined that the frames areappropriate for the frames to be provided with the content.

An information superimposition possible frame in which the exposed arearatio exceeds the reference value not only at one timing, but also at aplurality of timings may be determined to satisfy the condition 3.

The fact that the exposed area ratio is equal to or greater than thereference value indicates that a wide range of the informationsuperimposition possible frame can be viewed from the user even in acase where there is an object on the near side.

In this way, whether or not the frame is appropriate for the frame to beprovided with the content can be determined on the basis of thecondition 3 to select a frame that can be easily viewed when the contentis superimposed.

Determination of Condition 4 The appropriate visual field setting unit215 determines that the condition 4 is satisfied in a case where thestay time in the user visual field is equal to or greater than areference value.

FIG. 26 is a diagram illustrating an example of the stay time in theuser visual field.

As illustrated in FIG. 26 , it is assumed that the time for travelingfrom the vehicle position A to the vehicle position B is two seconds. Itis also assumed that the time for traveling from the vehicle position Bto the vehicle position C, the time for traveling from the vehicleposition C to the vehicle position D, and the time for traveling fromthe vehicle position D to the vehicle position E are two seconds each.These times are obtained on the basis of the distance between thevehicle positions and on the basis of the predicted traveling speed.

In a case where the timing of the vehicle passing through the vehicleposition A is set as a reference, the stay time in the user visual fieldof the information superimposition possible frame F_03 is obtained astime from the reference timing to the timing of traveling at a positionin front of the vehicle position C. Although the informationsuperimposition possible frame F_03 is not included in the visual fieldof the user at the vehicle position C indicated by the sector #3, theinformation superimposition possible frame F_03 is continuously includedin the visual field of the user up to the position in front of thevehicle position C. In this example, the stay time in the user visualfield of the information superimposition possible frame F_03 is 3.5seconds.

Similarly, the stay time in the user visual field of the informationsuperimposition possible frame F_04 is obtained as time from thereference timing to the timing of traveling at a position in front ofthe vehicle position D. Although the information superimpositionpossible frame F_04 is hardly included in the visual field of the userat the vehicle position D, the information superimposition possibleframe F_04 is continuously included in the visual field of the user upto the position in front of the vehicle position D. In the example, thestay time in the user visual field of the information superimpositionpossible frame F_04 is 5.5 seconds.

For example, in a case where the reference value of the minimum staytime in the user visual field is 3.0 seconds, the stay time of both ofthe information superimposition possible frame F_03 and the informationsuperimposition possible frame F_04 exceeds the reference value. Theinformation superimposition possible frame F_03 and the informationsuperimposition possible frame F_04 satisfy the condition 4, and it isdetermined that the frames are appropriate for the frames to be providedwith the content.

The reference value of the stay time may be set according to thetraveling speed. The reference value of 3.0 seconds described above isset in a case where, for example, the traveling speed is 5 m/sec (18km/h).

The fact that the stay time is equal to or greater than the referencevalue indicates that the information superimposition possible framecontinues to be in the visual field of the traveling user for a certaintime.

In this way, whether or not the frame is appropriate for the frame to beprovided with the content can be determined on the basis of thecondition 4 to select a frame that can be continuously viewed for acertain time when the content is superimposed.

Whether or not the information superimposition possible frame issuitable for the superimposition of the content is determined on thebasis of the conditions 1 to 4. Another condition, such as whether ornot the area of the information superimposition possible frame is equalto or greater than a reference value, may be used to determine whetheror not the information superimposition possible frame is suitable forthe superimposition of the content.

With reference again to FIG. 10 , the appropriate visual field settingunit 215 selects the information superimposition possible framessatisfying all of the conditions 1 to 4 as information superimpositionappropriate frames. The information superimposition possible framessatisfying at least any one of the conditions, instead of all of theconditions 1 to 4, may be selected as information superimpositionappropriate frames.

The appropriate visual field setting unit 215 sets an informationsuperimposition appropriate visual field on the basis of the informationsuperimposition appropriate frames. The information superimpositionappropriate visual field is a region of the visual field of the usersuitable for the superimposition of the content. For example, a regioninside a rectangle including all of the information superimpositionappropriate frames in the visual field of the user is set as theinformation superimposition appropriate visual field.

FIG. 27 is a diagram illustrating an example of setting the informationsuperimposition appropriate visual field.

As encircled by a thick line L21, a region inside a minimum rectanglesurrounding the information superimposition possible frame F_03 and theinformation superimposition possible frame F_04 selected as theinformation superimposition appropriate frames is set as the informationsuperimposition appropriate visual field. The informationsuperimposition possible frame F_03 and the information superimpositionpossible frame F_04 included in the information superimpositionappropriate visual field are used as the superimposition locations ofthe content.

Note that, in the example of FIG. 27 , the information superimpositionpossible frames included in the information superimposition appropriatevisual field are the information superimposition possible frame F_03 andthe information superimposition possible frame F_04 both determined tobe the information superimposition appropriate frames. The informationsuperimposition possible frames not determined to be the informationsuperimposition appropriate frames in the determination on the basis ofthe conditions 1 to 4 may also be used as the superimposition locationsof the content in a case where the frames are included in theinformation superimposition appropriate visual field.

In addition, the appropriate visual field setting unit 215 sets a regionoutside the information superimposition appropriate visual field as aninformation superimposition inappropriate visual field as illustrated inFIG. 27 . The information superimposition inappropriate visual field isnot used for the superimposition of the content and is used for displayof various messages, display of information representing the travelingstate such as the traveling speed, and the like. The display of thevelocity on the upper left illustrated in FIG. 2 and the like is displayusing the information superimposition inappropriate visual field.

The appropriate visual field setting unit 215 outputs information of allof the information superimposition appropriate frames included in theinformation superimposition appropriate visual field to thesuperimposition target frame selection unit 202.

<1-2-4. Process of Second Stage (Selection of InformationSuperimposition Appropriate Frame as Superimposition Location)>

Next, selection of the superimposition location executed by thesuperimposition target frame selection unit 202 in the process of thesecond stage will be described.

FIG. 28 is a block diagram illustrating a configuration example of thesuperimposition target frame selection unit 202 of FIG. 9 .

The superimposition target frame selection unit 202 includes apre-analysis unit 231, a fitness calculation unit 232, and asuperimposition location setting unit 233. The content acquired by thecontent acquisition unit 204 is input, to the pre-analysis unit 231, ascontent to be displayed. In addition, the information of the informationsuperimposition appropriate frames output from the informationsuperimposition appropriate visual field setting unit 201 is input tothe fitness calculation unit 232.

The pre-analysis unit 231 performs pre-analysis of content informationitems for all of the content to be displayed and sets display positiondetermination factors (Positioning Factors).

The content information items are metadata representing features of thecontent, and different items are provided for each type of content.Meanwhile, the display position determination factors are informationregarding specifications required for the display of the content and areused for selecting the information superimposition appropriate frame asthe superimposition location.

The pre-analysis unit 231 classifies each content into any one ofcontent types including “moving image,” “still image,” and “character(text).” The classification of the content type is performed on thebasis of, for example, an extension of the file of the content.

FIG. 29 depicts diagrams each illustrating an example of the contentinformation items. Overlapping description will be appropriatelyomitted.

A of FIG. 29 illustrates content information items of text content witha file name “File_01.”

As illustrated in A of FIG. 29 , the pre-analysis unit 231 sets items“Type,” “Number of Letters,” “Time for Reading,” “Proportion,” and “Timefor Viewing” as the content information items of the text content.

“Type” represents the content type.

“Number of Letters” represents the number of letters included in thecontent.

“Time for Reading” represents the reading time. The reading time is setaccording to, for example, the number of letters.

“Proportion” represents the aspect ratio of the region required fordisplaying the content.

“Time for Viewing” represents the required time for viewing. In the caseof the text content, the required time for viewing is set to the sametime as the reading time.

In the example of A in FIG. 29 , “Number of Letters” is set to 42. Inaddition, “Time for Reading” is set to 10 seconds. “Proportion” is setto 4:3. “Time for Viewing” is set to 10 seconds which is the same as“Time for Reading.”

In this case, the pre-analysis unit 231 sets “Proportion” of 4:3 and“Time for Viewing” of 10 seconds as the display position determinationfactors of the text content of “File_01.”

B of FIG. 29 illustrates content information items of moving imagecontent with a file name “File_02.”

As illustrated in B of FIG. 29 , the pre-analysis unit 231 sets items“Type,” “Playback Duration,” “Proportion,” and “Time for Viewing” as thecontent information items of the moving image content.

“Playback Duration” represents the reproduction time.

“Proportion” represents the aspect ratio of each frame included in themoving image content.

“Time for Viewing” represents the required time for viewing. In the caseof the moving image content, the required time for viewing is set to thesame time as the reproduction time.

In the example of B in FIG. 29 , “Playback Duration” is set to 20seconds. In addition, “Proportion” is set to 16:9. “Time for Viewing” isset to 20 seconds which is the same as “Playback Duration.”

In this case, the pre-analysis unit 231 sets “Proportion” of 16:9 and“Time for Viewing” of 20 seconds as the display position determinationfactors of the moving image content of “File_02.” C of FIG. 29illustrates content information items of still image content with a filename “File_03,” and D of FIG. 29 illustrates content information itemsof still image content with a file name “File_04.”

As illustrated in C and D of FIG. 29 , the pre-analysis unit 231 setsitems “Type,” “Text,” “Number of Letters,” “Time for Reading,”“Proportion,” and “Time for Viewing” as the content information items ofthe still image content.

“Text” indicates whether or not text elements are included in the image.For the content of “still image,” the pre-analysis unit 231 analyzes theimage to identify whether the image is a “still image” with textelements or a “still image” without text elements. The other contentinformation items of the still image content with text elements are thesame as the content information items of the text content.

In the example of C in FIG. 29 , “Text” is set as an image with textelements (Included). The still image content of “File_03” is still imagecontent with text elements. In addition, “Number of Letters” is set to28. “Time for Reading” is set to 7 seconds. “Proportion” is set to 3:4.“Time for Viewing” is set to 7 seconds which is the same as “Time forReading.”

In this case, the pre-analysis unit 231 sets “Proportion” of 3:4 and“Time for Viewing” of 7 seconds as the display position determinationfactors of the still image content of “File_03.”

Meanwhile, “Text” is set as an image with text elements in the exampleof D in FIG. 29 . The still image content of “File_04” is also stillimage content with text elements. In addition, “Number of Letters” isset to 18. “Time for Reading” is set to 5 seconds. “Proportion” is setto 1:1. “Time for Viewing” is set to 5 seconds which is the same as“Time for Reading.”

In this case, the pre-analysis unit 231 sets “Proportion” of 1:1 and“Time for Viewing” of 5 seconds as the display position determinationfactors of the still image content of “File_04.”

The pre-analysis unit 231 outputs the information of the displayposition determination factors of each content set in this way to thefitness calculation unit 232.

The fitness calculation unit 232 calculates the fitness of eachcombination of the content after the pre-analysis by the pre-analysisunit 231 and the information superimposition appropriate frame includedin the information superimposition appropriate visual field. That is,the fitness of each of all of the contents and each of the informationsuperimposition appropriate frames included in the informationsuperimposition appropriate visual field is calculated.

For example, the fitness calculation unit 232 specifies the aspect ratioand the stay time in the user visual field of each informationsuperimposition appropriate frame on the basis of, for example, theinformation supplied from the information superimposition appropriatevisual field setting unit 201.

The fitness calculation unit 232 compares the specified aspect ratio andstay time in the user visual field with the display positiondetermination factors of the content to calculate the fitness of thecontent and the information superimposition appropriate frame. Asdescribed above, the aspect ratio (“Proportion”) and the required timefor viewing (“Time for Viewing”) are set as the display positiondetermination factors.

FIG. 30 depicts diagrams each illustrating an example of the fitness.

Here, it is assumed that four information superimposition appropriateframes including information superimposition appropriate frames F_11 toF_14 are included in the information superimposition appropriate visualfield. Frame_11 to Frame_14 illustrated in FIG. 30 represent theinformation superimposition appropriate frames F_11 to F_14,respectively. It is assumed that the aspect ratios of the informationsuperimposition appropriate frames F_11 to F_14 are 1:0.9, 3:4.2, 4:3.2,and 16:8, respectively, and the stay time in the user visual field is 4seconds, 7 seconds, 14 seconds, and 17 seconds each.

A of FIG. 30 illustrates the fitness of the text content of “File_01”and each of the information superimposition appropriate frames F_11 toF_14. The display position determination factors of the text content of“File_01” are the aspect ratio of 4:3 and the required time for viewingof 10 seconds as described above.

As illustrated in A of FIG. 30 , the pre-analysis unit 231 determinesthat the fitness of the aspect ratio is 20 on the basis of the aspectratio of 4:3 for the text content and the aspect ratio of 1:0.9 for theinformation superimposition appropriate frame F_11. For example, thecloser the ratios, the higher the obtained fitness.

In addition, the pre-analysis unit 231 determines that the fitness ofthe required time for viewing is 10 on the basis of the required timefor viewing of 10 seconds for the text content and the stay time in theuser visual field of 4 seconds for the information superimpositionappropriate frame F_11. For example, the closer the required time andthe stay time, the higher the obtained fitness.

The pre-analysis unit 231 adds the fitness of the aspect ratio and thefitness of the required time for viewing to determine that the overallfitness of the text content of “File_01” and the informationsuperimposition appropriate frame F_11 is 30.

Although the fitness of the aspect ratio and the fitness of the requiredtime for viewing are added to obtain the overall fitness in thisexample, the method of obtaining the overall fitness is arbitrary, suchas conducting the addition after weighting the fitness of one of them.

The pre-analysis unit 231 similarly calculates the fitness of each ofthe information superimposition appropriate frames F_12 to F_14 and thetext content of “File_01.”

In the example of A in FIG. 30 , the overall fitness of the text contentof “File_01” and the information superimposition appropriate frame F_12is 30, and the overall fitness of the text content of “File_01” and theinformation superimposition appropriate frame F_13 is 90. In addition,the overall fitness of the text content of “File_01” and the informationsuperimposition appropriate frame F_14 is 70.

The pre-analysis unit 231 similarly calculates the fitness of each ofthe moving image content of “File_02,” the still image content of“File_03,” and the still image content of “File_04” and each of theinformation superimposition appropriate frames F_11 to F_14.

B of FIG. 30 illustrates the fitness of the moving image content of“File_02” and each of the information superimposition appropriate framesF_11 to F_14. The display position determination factors of the movingimage content of “File_02” are the aspect ratio of 16:9 and the requiredtime for viewing of 20 seconds as described above.

In the example of B in FIG. 30 , the overall fitness of the moving imagecontent of “File_02” and the information superimposition appropriateframe F_11 is 0, and the overall fitness of the moving image content of“File_02” and the information superimposition appropriate frame F_12 is0. In addition, the overall fitness of the moving image content of“File_02” and the information superimposition appropriate frame F_13 is50, and the overall fitness of the moving image content of “File_02” andthe information superimposition appropriate frame F_14 is 80.

C of FIG. 30 illustrates the fitness of the still image content of“File_03” and each of the information superimposition appropriate framesF_11 to F_14. The display position determination factors of the stillimage content of “File_03” are the aspect ratio of 3:4 and the requiredtime for viewing of 7 seconds as described above.

In the example of C in FIG. 30 , the overall fitness of the still imagecontent of “File_03” and the information superimposition appropriateframe F_11 is 30, and the overall fitness of the still image content of“File_03” and the information superimposition appropriate frame F_12 is90. In addition, the overall fitness of the still image content of“File_03” and the information superimposition appropriate frame F_13 is60, and the overall fitness of the still image content of “File_03” andthe information superimposition appropriate frame F_14 is 50.

D of FIG. 30 illustrates the fitness of the still image content of“File_04” and each of the information superimposition appropriate framesF_11 to F_14. The display position determination factors of the stillimage content of “File_04” are the aspect ratio of 1:1 and the requiredtime for viewing of 5 seconds as described above.

In the example of D in FIG. 30 , the overall fitness of the still imagecontent of “File_04” and the information superimposition appropriateframe F_11 is 70, and the overall fitness of the still image content of“File_04” and the information superimposition appropriate frame F_12 is70. In addition, the overall fitness of the still image content of“File_04” and the information superimposition appropriate frame F_13 is70, and the overall fitness of the still image content of “File_04” andthe information superimposition appropriate frame F_14 is 60.

The fitness calculation unit 232 outputs the information of the fitnesscalculated in this way to the superimposition location setting unit 233.

The superimposition location setting unit 233 sets the informationsuperimposition appropriate frame as the superimposition location ofeach content according to the fitness calculated by the fitnesscalculation unit 232. For example, the superimposition location settingunit 233 sequentially allocates the information superimpositionappropriate frames with high fitness to set the informationsuperimposition appropriate frames as the superimposition locations.

FIG. 31 is a diagram illustrating an example of setting thesuperimposition locations.

The left side of FIG. 31 illustrates an example of calculating thefitness. Each fitness is appropriately ranked in a range from “Not Good”to “Very Good.”

A table illustrated beyond an outline arrow is a table illustrating theoverall fitness of each of the text content of “File_01,” the movingimage content of “File_02,” the still image content of “File_03,” andthe still image content of “File_04” and each of the informationsuperimposition appropriate frames F_11 to F_14.

As encircled and indicated by an ellipse, the superimposition locationsetting unit 233 sets the information superimposition appropriate frameF_13 with the highest fitness among the information superimpositionappropriate frames F_11 to F_14 as the superimposition location of thetext content of “File_01.”

In addition, the superimposition location setting unit 233 sets theinformation superimposition appropriate frame F_14 with the highestfitness among the information superimposition appropriate frames F_11 toF_14 as the superimposition location of the moving image content of“File_02.”

The superimposition location setting unit 233 sets the informationsuperimposition appropriate frame F_12 with the highest fitness amongthe information superimposition appropriate frames F_11 to F_14 as thesuperimposition location of the still image content of “File_03.”

The superimposition location setting unit 233 sets the informationsuperimposition appropriate frame F_11 with the highest fitness amongthe information superimposition appropriate frames F_11 to F_14 as thesuperimposition location of the still image content of “File_04.”

Note that, although the fitness of the still image content of “File_04”and each of the information superimposition appropriate frames F_11,F_12, and F_13 is 70, the information superimposition appropriate frameF_11 not set as the superimposition location of any content is set here.The information superimposition appropriate frame F_12 is already set asthe superimposition location of the still image content of “File_03” onthe basis of the fitness higher than 70, and the informationsuperimposition appropriate frame F_13 is already set as thesuperimposition location of the text content of “File_01” on the basisof the fitness higher than 70.

FIG. 32 is a diagram illustrating an example of the superimpositionlocations set in this way.

Buildings B101 to B103 are in a scene image illustrated in FIG. 32 . Inthe example of FIG. 32 , the information superimposition appropriateframe F_11 and the information superimposition appropriate frame F_12described above are the information superimposition appropriate framesset for the building B101 on the left side. The informationsuperimposition appropriate frame F_13 is the informationsuperimposition appropriate frame set for the building B102 at thecenter. The information superimposition appropriate frame F_14 is theinformation superimposition appropriate frame set for the building B103on the right side.

In the case where the information superimposition appropriate frame asthe superimposition location is set in this way, the still image contentof “File_04,” in which the calculated fitness with the informationsuperimposition appropriate frame F_11 is 70, is superimposed on theinformation superimposition appropriate frame F_11. Furthermore, thestill image content of “File_03,” in which the calculated fitness withthe information superimposition appropriate frame F_12 is 80, issuperimposed on the information superimposition appropriate frame F_12.

The text content of “File_01,” in which the calculated fitness with theinformation superimposition appropriate frame F_13 is 90, issuperimposed on the information superimposition appropriate frame F_13.The moving image content of “File_02,” in which the calculated fitnesswith the information superimposition appropriate frame F_14 is 80, issuperimposed on the information superimposition appropriate frame F_14.

The superimposition location setting unit 233 outputs the information ofthe superimposition location of each content set in this way, that is,information indicating which information superimposition appropriateframe is to be provided with each content, to the display control unit203.

Although the fitness of the content and the information superimpositionappropriate frame is calculated on the basis of the aspect ratio and thetime of the content and the information superimposition appropriateframe in the description above, the fitness may be calculated on thebasis of other elements. For example, the fitness may be calculated onthe basis of only the aspect ratio or may be calculated on the basis ofonly the time (required time for viewing and stay time in the visualfield).

That is, the fitness of the content and the information superimpositionappropriate frame can be calculated on the basis of at least any one ofthe display position determination factors.

Furthermore, the moving image content may be superimposed on theinformation superimposition appropriate frame at a dark place, and thetext content may be superimposed on the information superimpositionappropriate frame at a bright place. In this way, the fitness may becalculated on the basis of other elements. In this case, the fitness iscalculated on the basis of the luminance of the informationsuperimposition appropriate frame and the type of content.

The fitness may be calculated so as to superimpose the moving imagecontent on the information superimposition appropriate frame at a longdistance and superimpose the text content on the informationsuperimposition appropriate frame at a short distance. In this case, thefitness is calculated on the basis of the distance to the informationsuperimposition appropriate frame and the type of content.

In this way, various methods can be adopted for the method ofcalculating the fitness of each content and the informationsuperimposition appropriate frame.

Although the analysis of the content information items and the displayposition determination factors is performed by the vehicle controlsystem 100, the server side that manages the content may perform theanalysis. The server provides the vehicle control system 100 with theinformation regarding the content information items and the displayposition determination factors in association with the content.

<1-2-5. Process of Third State (Execution of Display)>

Next, a display process executed by the display control unit 203 in theprocess of the third stage will be described.

FIG. 33 is a block diagram illustrating a configuration example of thedisplay control unit 203 of FIG. 9 .

The display control unit 203 includes a fitting contrast calculationunit 251, a contrast adjustment unit 252, a mask processing unit 253,and a display processing unit 254.

The analysis results of the scene image output from the informationsuperimposition appropriate visual field setting unit 201 and theinformation regarding the information superimposition appropriate frameas the superimposition location of each content output from thesuperimposition target frame selection unit 202 are input to the fittingcontrast calculation unit 251. The content output from the contentacquisition unit 204 is input to the contrast adjustment unit 252, andthe scene image is input to the mask processing unit 253.

The fitting contrast calculation unit 251 calculates fitting contrast ofeach information superimposition appropriate frame on the basis of thecontext at the location of the information superimposition appropriateframe that is set as the superimposition location of the content. Thefitting contrast is used to adjust the contrast of the content such thatthe appearance of the content becomes the same as the actual appearanceat the location of the information superimposition appropriate frame.

The fitting contrast calculation unit 251 calculates the fittingcontrast on the basis of, for example at least any one of the state ofthe sunlight, the state of the lighting, the state of the atmosphere,and the distance from the vehicle position. Although at least any one ofthe state of the sunlight, the state of the lighting, the state of theatmosphere, and the distance from the vehicle position is used as thecontext in this example, other states, such as weather and temperature,may be used as the context.

For example, the state of the sunlight, the state of the lighting, thestate of the atmosphere, and the distance from the vehicle position areincluded in the analysis results of the scene image. The distance fromthe vehicle position to the location of the information superimpositionappropriate frame may be acquired on the basis of detection results of adistance sensor.

The fitting contrast calculation unit 251 outputs the informationregarding the fitting contrast of each information superimpositionappropriate frame to the contrast adjustment unit 252.

The contrast adjustment unit 252 adjusts the contrast of the contentprovided with the superimposition location according to the fittingcontrast of the information superimposition appropriate frame as thesuperimposition location. The contrast adjustment unit 252 outputs thecontent after the adjustment of the contrast to the display processingunit 254.

FIG. 34 is a diagram illustrating an example of the adjustment of thecontrast on the basis of the fitting contrast.

The information superimposition appropriate frames F_11 to F_14illustrated in FIG. 34 are the same as the information superimpositionappropriate frames described with reference to FIG. 32 .

In the example of FIG. 34 , the information superimposition appropriateframe F_11 is set on the wall surface of the building B101 at a shortdistance. The sunlight shines on the wall surface provided with theinformation superimposition appropriate frame F_11, and the state of thesunlight of the information superimposition appropriate frame F_11 is asunny state. The state of the sunlight is specified on the basis of, forexample, the luminance of the wall surface provided with the informationsuperimposition appropriate frame F_11.

In this case, as illustrated at the tip of an extension line in FIG. 34, the fitting contrast calculation unit 251 determines that the fittingcontrast of the information superimposition appropriate frame F_11 is100% on the basis of the fact that the information superimpositionappropriate frame F_11 is at a short distance and the state of thesunlight is a sunny state.

Since the fitting contrast is 100%, the still image content of “File_04”with the superimposition location at the information superimpositionappropriate frame F_11 is superimposed on the informationsuperimposition appropriate frame F_11 at the original contrast withoutthe adjustment of the contrast.

Meanwhile, the state of the sunlight of the information superimpositionappropriate frame F_12 set on the wall surface of the building B101 at ashort distance is a little shady state.

In this case, the fitting contrast calculation unit 251 determines thatthe fitting contrast of the information superimposition appropriateframe F_12 is 60% on the basis of the fact that the informationsuperimposition appropriate frame F_12 is at a short distance and thestate of the sunlight is a little shady state.

Since the fitting contrast is 60%, the contrast of the still imagecontent of “File_03” with the superimposition location at theinformation superimposition appropriate frame F_12 is reduced to 60%,and the content is superimposed on the information superimpositionappropriate frame F_12 in this state.

In a case where the actual appearance of the wall surface provided withthe information superimposition appropriate frame F_11 and the actualappearance of the wall surface provided with the informationsuperimposition appropriate frame F_12 are compared, the wall surfaceprovided with the information superimposition appropriate frame F_12 isshady, so that the wall surface looks darker with lower contrast. Thecontrast can be reduced according to the actual appearance of the wallsurface to display the still image content of “File_03” in a state thatthe content is fit into the appearance of the wall surface provided withthe content.

The information superimposition appropriate frame F_13 is set on thewall surface of the building B102 at a long distance. The state of theatmosphere of the information superimposition appropriate frame F_13 isin a foggy state. The foggy state of the atmosphere is specified on thebasis of, for example, the luminance and the contrast of the wallsurface provided with the information superimposition appropriate frameF_13.

In this case, the fitting contrast calculation unit 251 determines thatthe fitting contrast of the information superimposition appropriateframe F_13 is 10% on the basis of the fact that the informationsuperimposition appropriate frame F_13 is at a long distance and thestate of the atmosphere is a foggy state.

Since the fitting contrast is 10%, the contrast of the text content of“File_01” with the superimposition location at the informationsuperimposition appropriate frame F_13 is reduced to 10%, and thecontent is superimposed on the information superimposition appropriateframe F_13 in this state.

In a case where the actual appearance of the wall surface provided withthe information superimposition appropriate frame F_11 and the actualappearance of the wall surface provided with the informationsuperimposition appropriate frame F_13 are compared, the contrast of thewall surface provided with the information superimposition appropriateframe F_13 looks lower because the wall surface is at a long distanceand looks blurry due to the fog. The contrast can be reduced accordingto the actual appearance of the wall surface to display the text contentof “File_01” in a state that the content is fit into the appearance ofthe wall surface provided with the content.

The information superimposition appropriate frame F_14 is set on thewall surface of the building B103 at a short distance. The state of thesunlight of the information superimposition appropriate frame F_14 is asunny state.

In this case, the fitting contrast calculation unit 251 determines thatthe fitting contrast of the information superimposition appropriateframe F_14 is 100% on the basis of the fact that the informationsuperimposition appropriate frame F_14 is at a short distance and thestate of the sunlight is a sunny state as in the case of the informationsuperimposition appropriate frame F_11.

Since the fitting contrast is 100%, the moving image content of“File_02” with the superimposition location at the informationsuperimposition appropriate frame F_14 is superimposed on theinformation superimposition appropriate frame F_14 at the originalcontrast without the adjustment of the contrast.

In this way, the contrast of each content is adjusted on the basis ofthe fitting contrast calculated according to the actual appearance ofthe location of the information superimposition appropriate frame as thesuperimposition location. This allows the user to view the content as ifthe content is displayed on the wall surface of the actual building.

With reference again to FIG. 33 , in the case where the front object,such as a tree and a telephone pole, covers the informationsuperimposition appropriate frame, the mask processing unit 253 executesa masking process to cut out the image of the front object from thescene image.

The masking process of the mask processing unit 253 is executed in acase where there is a front object overlapping the informationsuperimposition appropriate frame. The mask processing unit 253 outputs,to the display processing unit 254, a part image that is the image ofthe front object cut out by executing the masking process.

The display processing unit 254 controls the projection unit 106A tosuperimpose each content on the information superimposition appropriateframe that is set as the superimposition location of each content.

In addition, the display processing unit 254 displays the part imagesupplied from the mask processing unit 253 on top of the content. Thepart image is displayed on top of the position where the front object isactually viewed. The display of the content and the like of the displayprocessing unit 254 is repeated during the traveling of the vehicle.

FIG. 35 is a diagram illustrating an example of the display of the frontobject.

As illustrated on the left end of FIG. 35 , it is assumed that a tree infront of a building is covering the information superimpositionappropriate frame set on the wall surface of the building at a timing oftraveling at a certain position. A range indicated by a broken line onthe left end of FIG. 35 is a range of the information superimpositionappropriate frame. In this case, a colored range illustrated beyond anoutline arrow #31 is a range where the front object and the informationsuperimposition appropriate frame overlap.

The mask processing unit 253 detects an external form of the frontobject and generates a mask image M as illustrated beyond an arrow #41.In addition, the mask processing unit 253 uses the mask image M to applya masking process to the scene image to cut out a part image P of thefront object.

As illustrated beyond an outline arrow #32, the display processing unit254 superimposes the content on the information superimpositionappropriate frame.

As described above, the display of content is performed by displayingthe content on the windshield G, and as for the appearance viewed by theuser, the user views the content on top of the front object as indicatedbeyond the outline arrow #32. Diagonal lines provided over the entireinformation superimposition appropriate frame illustrated beyond theoutline arrow #32 indicate that the entire content can be viewed withoutbeing covered by the front object.

The display processing unit 254 displays the part image P on top of theposition of the front object to realize the display as if the content isdisplayed on the wall surface of the building behind the front object asillustrated beyond an outline arrow #33.

If the part image P is not displayed on top of the content, the contentis viewed in front of the front object as illustrated beyond the outlinearrow #32, and the content cannot be viewed as if the content isdisplayed on the wall surface of the building. The part image P can bedisplayed on top of the content to realize the appearance as if thecontent is displayed on the wall surface of the actual building withouta sense of discomfort.

<<1-3. Motion of Vehicle Control System>>

Here, the motion of the vehicle control system 100 configured in thisway will be described.

<1-3-1. Information Display Process>

First, an information display process as an overall process will bedescribed with reference to a flow chart of FIG. 36 .

In step S1, the automatic drive control unit 112 controls the cameraincluded in the data acquisition unit 102 to cause the camera to startimaging the scene in the traveling direction of the vehicle. The sceneimage obtained by imaging is supplied to the output control unit 105.

In step S2, the automatic drive control unit 112 controls the sensorsincluded in the data acquisition unit 102 to cause various sensors tostart the measurement. The prediction or the like of the state of thevehicle is performed on the basis of the measurement results and thelike of the sensors. The information regarding the state of the vehicleis supplied as progress state prediction information to the outputcontrol unit 105.

In step S3, the information superimposition appropriate visual fieldsetting unit 201 executes the information superimposition appropriatevisual field setting process that is the process of the first stage. Asa result of the information superimposition appropriate visual fieldsetting process, the information superimposition appropriate framessuitable for the superimposition of the content are selected. Details ofthe information superimposition appropriate visual field setting processwill be described later with reference to flow charts of FIGS. 37 and 38.

In step S4, the superimposition target frame selection unit 202 executesthe superimposition target frame selection process that is the processof the second stage. As a result of the superimposition target frameselection process, the information superimposition appropriate frame asthe superimposition location is set from among the informationsuperimposition appropriate frames included in the informationsuperimposition appropriate visual field. Details of the superimpositiontarget frame selection process will be described later with reference toa flow chart of FIG. 39 .

In step S5, the display control unit 203 executes the display processthat is the process of the third stage. As a result of the displayprocess, the content is displayed. Details of the display process willbe described later with reference to a flow chart of FIG. 40 .

The processes of steps S3 to S5 are repeated during the traveling of thevehicle.

<1-3-2. Information Superimposition Appropriate Visual Field SettingProcess>

The information superimposition appropriate visual field setting processexecuted in step S3 of FIG. 36 will be described with reference to theflow charts of FIGS. 37 and 38.

In step S11, the image analysis unit 211 of the informationsuperimposition appropriate visual field setting unit 201 analyzes thescene image. For example, the outlines of the buildings in the sceneimage are detected, and the brightness of the surroundings is detected.

In step 512, the light state mode setting unit 212 sets any one of thelight state modes including the “daytime mode,” the “dusk mode,” and the“night mode” on the basis of the brightness of the surroundings.

In step 513, the object detection unit 213 determines the position ofeach object. That is, the object detection unit 213 plots the buildings,for which the outlines are detected, on the map and determines thepositions of the buildings in the scene image. In addition, the objectdetection unit 213 determines the positions of the objects, such astrees, in addition to the buildings. The object detection unit 213generates a three-dimensional mode of the objects for which thepositions are determined.

In step 514, the frame setting unit 214 sets the informationsuperimposition possible frames on the surfaces of the objects.

In step 515, the frame setting unit 214 excludes the inappropriateinformation superimposition possible frames according to the light statemode. Here, the information superimposition possible frames that lookunnatural in the case where the content is superimposed are excluded ina manner as described above on the basis of the exclusion rules 1 to 4.The information of the information superimposition possible frames thatare not excluded is supplied to the appropriate visual field settingunit 215.

In step S16, the appropriate visual field setting unit 215 focuses onone information superimposition possible frame.

In step 517, the appropriate visual field setting unit 215 calculatesthe angular velocity in the user visual field of the focused informationsuperimposition possible frame.

In step S18, the appropriate visual field setting unit 215 determineswhether or not the calculated angular velocity in the user visual fieldis in the reference range.

In a case where the appropriate visual field setting unit 215 determinesthat the angular velocity in the user visual field is in the referencerange in step S18, the appropriate visual field setting unit 215calculates the angle of the focused information superimposition possibleframe with respect to the vehicle traveling direction in step S19.

In step S20, the appropriate visual field setting unit 215 determineswhether or not the calculated angle with respect to the vehicletraveling direction is equal to or greater than the minimum referencevalue.

In a case where the appropriate visual field setting unit 215 determinesthat the angle with respect to the vehicle traveling direction is equalto or greater than the minimum reference value in step S20, theappropriate visual field setting unit 215 calculates the exposed arearatio of the focused information superimposition possible frame in stepS21.

In step S22, the appropriate visual field setting unit 215 determineswhether or not the calculated exposed area ratio is equal to or greaterthan the reference value.

In a case where the appropriate visual field setting unit 215 determinesthat the exposed area ratio is equal to or greater than the referencevalue in step S22, the appropriate visual field setting unit 215calculates the stay time in the user's visual field of the focusedinformation superimposition possible frame in step S23.

In step S24, the appropriate visual field setting unit 215 determineswhether or not the calculated stay time in the user's visual field isequal to or greater than the reference value.

In a case where the appropriate visual field setting unit 215 determinesthat the stay time is equal to or greater than the reference value instep S24, the appropriate visual field setting unit 215 sets the focusedinformation superimposition possible frame as the informationsuperimposition appropriate frame in step S25.

Conversely, in a case where the appropriate visual field setting unit215 determines that the angular velocity in the user visual field is notin the reference range in step S18, the appropriate visual field settingunit 215 sets the focused information superimposition possible frame asa frame inappropriate for the superimposition of information in stepS26. This is similar in a case where the appropriate visual fieldsetting unit 215 determines that the angle with respect to the vehicletraveling direction is not equal to or greater than the minimumreference value in step S20, in a case where the appropriate visualfield setting unit 215 determines that the exposed area ratio is notequal to or greater than the reference value in step S22, or in a casewhere the appropriate visual field setting unit 215 determines that thestay time in the user's visual field is not equal to or greater than thereference value in step S24.

After the process of step S25 or step S26, the appropriate visual fieldsetting unit 215 determines whether or not all of the informationsuperimposition possible frames are focused in step S27.

In a case where the appropriate visual field setting unit 215 determinesthat there are information superimposition possible frames not focusedyet in step S27, the appropriate visual field setting unit 215 returnsto step S16 and focuses on another information superimposition possibleframe to repeat the process. In a case where the appropriate visualfield setting unit 215 determines that all of the informationsuperimposition possible frames are focused in step S27, the processproceeds to step S28.

In step S28, the appropriate visual field setting unit 215 sets, as theinformation superimposition appropriate visual field, the region insidethe rectangle including all of the information superimpositionappropriate frames. The process then returns to step S3 of FIG. 36 , andthe subsequent process is executed.

<1-3-3. Superimposition Target Frame Selection Process>

The superimposition target frame selection process executed in step S4of FIG. 36 will be described with reference to the flow chart of FIG. 39.

Note that the pre-analysis of the content is performed by thepre-analysis unit 231 in parallel with the process of FIG. 36 . Thedisplay position determination factors obtained in the pre-analysisprocess are supplied from the pre-analysis unit 231 to the fitnesscalculation unit 232.

In step S41, the fitness calculation unit 232 of the superimpositiontarget frame selection unit 202 focuses on one content to be displayed.

In step S42, the fitness calculation unit 232 calculates the fitness ofthe focused content and each of all of the information superimpositionappropriate frames in the information superimposition appropriate visualfield. The fitness is calculated with reference to, for example, theaspect ratio, the required time for viewing, and the like included inthe display position determination factors as described above.

In step S43, the fitness calculation unit 232 determines whether or notall of the contents are focused.

In a case where the fitness calculation unit 232 determines that thereare contents not focused yet in step S43, the fitness calculation unit232 returns to step S41 and focuses on another content to repeat theprocess. As a result, the fitness of each content and each of all of theinformation superimposition appropriate frames is calculated.

In a case where the fitness calculation unit 232 determines that all ofthe contents are focused in step S43, the superimposition locationsetting unit 233 sets the information superimposition appropriate frameas the superimposition location of each content on the basis of thefitness in step S44. The process then returns to step S4 of FIG. 36 ,and the subsequent process is executed.

<1-3-4. Display Process>

The display process executed in step S5 of FIG. 36 will be describedwith reference to the flow chart of FIG. 40 .

In step S51, the fitting contrast calculation unit 251 of the displaycontrol unit 203 focuses on one content provided with thesuperimposition location.

In step S52, the fitting contrast calculation unit 251 calculates thefitting contrast on the basis of the context at the location of theinformation superimposition appropriate frame as the superimpositionlocation.

In step S53, the contrast adjustment unit 252 adjusts the contrast ofthe focused content on the basis of the fitting contrast.

In step S54, the mask processing unit 253 executes the masking processto cut out the image of the front object in a case where there is anobject in front of the information superimposition appropriate frame asthe superimposition location of the focused content.

In step S55, the display processing unit 254 superimposes the content onthe information superimposition appropriate frame.

In step S56, the display processing unit 254 appropriately displays theimage of the front object on top of the content.

In step S57, the display control unit 203 determines whether or not allof the contents are focused.

In a case where the display control unit 203 determines that there arecontents not focused yet in step S57, the display control unit 203returns to step S51 and focuses on another content to repeat theprocess.

In a case where the display control unit 203 determines that all of thecontents are focused in step S57, the display control unit 203 returnsto step S5 of FIG. 36 and executes the subsequent process. As a result,all of the contents are superimposed on the information superimpositionappropriate frames set as the superimposition locations.

As a result of the series of processes described above, the content canbe displayed at optimal places in the scene, in a form that the contentis fit into the scene. The user can view the content in a way similar toviewing a normal scene.

<1-3-5. Pre-Analysis Process>

The pre-analysis process of the content will be described with referenceto a flow chart of FIG. 41 . The pre-analysis process is appropriatelyexecuted in parallel with the process of FIG. 36 .

In step S101, the pre-analysis unit 231 of the superimposition targetframe selection unit 202 focuses on one content to be displayed amongthe contents supplied from the content acquisition unit 204.

In step S102, the pre-analysis unit 231 classifies the focused contentaccording to the content type on the basis of the extension of the file.

In a case where the content type of the focused content is “movingimage,” the pre-analysis unit 231 sets the aspect ratio and the requiredtime for viewing as the display position determination factors in stepS103.

In a case where the content type of the focused content is “stillimage,” the pre-analysis unit 231 analyzes the focused still imagecontent to determine whether or not there is a text element in stepS104.

In step S105, the pre-analysis unit 231 determines whether or not a textelement is included, and in a case where the pre-analysis unit 231determines that a text element is not included, the pre-analysis unit231 sets the aspect ratio as the display position determination factorin step S106.

Conversely, in a case where the pre-analysis unit 231 determines that atext element is included in step S105, the pre-analysis unit 231 setsthe aspect ratio and the required time for viewing as the displayposition determination factors in step S107.

In a case where the content type of the focused content is “character,”the pre-analysis unit 231 sets the aspect ratio and the required timefor viewing as the display position determination factors in step S108.

After setting the display position determination factors in steps S103,S106, and S107 or in step S108, the pre-analysis unit 231 determineswhether or not all of the contents are focused in step S109.

In a case where the pre-analysis unit 231 determines that there arecontents not focused yet in step S109, the pre-analysis unit 231 returnsto step S101 and focuses on another content to repeat the process.

In a case where the pre-analysis unit 231 determines that all of thecontents are focused in step S109, the pre-analysis process is finished.The information regarding the display position determination factors ofeach content is supplied from the pre-analysis unit 231 to the fitnesscalculation unit 232 and used for calculating the fitness.

Note that the process of each step described above may be appropriatelyexecuted in parallel with another process, or the process may beexecuted before or after another process by changing an order.

<<1-4. Modification>>

Although the moving body that the user is riding is a car in thedescription above, the moving body may be other moving bodies, such as abicycle, a motorcycle, a train, and an airplane. The technique describedabove can be applied to the display of content in various moving bodiesas long as a device serving as a display surface of the content isprepared for the moving body between the object and the user.

In addition, the technique can also be applied to the display of contentin a case where the user wears a transmissive HMD and moves on foot. Inthis case, the HMD displays the content on top of the scene in thetraveling direction.

Although different contents are superimposed in a case where there is aplurality of contents to be superimposed on the scene at a certaintiming, one type of content may be superimposed on a plurality ofinformation superimposition appropriate frames.

Although the image processing applied to the content on the basis of thecontext at the location of the information superimposition appropriateframe is the adjustment of the contrast, other processing, such asadjustment of luminance and adjustment of color, may be executed.

2. Second Embodiment <<2-1. Summary of Second Embodiment>>

In the example described above, the information superimpositionappropriate frames are set on the basis of the scene image in thetraveling direction of the vehicle imaged by the camera included in thedata acquisition unit 102, and the content is superimposed and displayedon the set information superimposition appropriate frames.

However, the example is on the basis of the scene image captured by thecamera, and the visual point of the user is not taken into account.Therefore, the appearance may become unnatural.

That is, the content existing in the visual line direction of the usershould be in focus and clearly viewed, and the content out of the visualline direction should be defocused and viewed in a blurred state.However, this is not considered in the method described above. All ofthe contents are displayed such that the entire contents are clearlyviewed regardless of the visual line direction, and the appearance maybecome unnatural.

Therefore, the user riding on the vehicle may be imaged to detect thevisual point position of the user, and the content superimposed on theinformation superimposition appropriate frame according to the visualpoint position may be displayed by applying a defocusing process to thecontent such that the farther the position of the content from thevisual point position, the more obscure the view of the content.

FIG. 42 depicts diagrams each describing a summary of a secondembodiment of the vehicle control system 100 that displays the contentby applying a defocusing process to the content to be superimposed onthe information superimposition appropriate frame according to a visualpoint position EP of the user riding on the vehicle.

That is, the vehicle control system 100 of FIG. 42 is provided with acamera 271 that images a scene image in the traveling direction of thevehicle and a camera 272 that images a user 273 as an occupant insidethe vehicle.

Furthermore, the output control unit 105 executes the series ofprocesses described above to superimpose the content on the informationsuperimposition appropriate frame on the basis of the scene imagecaptured by the camera 271 and applies a defocusing process to thecontent to be displayed on the projection unit 106A according to thedistance from the visual point position EP on the projection unit 106Aof the user 273 inside the vehicle imaged by the camera 272.

Here, the defocusing denotes processing executed in displaying theimage, in which the image is processed into a state that the image isout of focus such that the image is obscurely displayed to the user.

That is, in the second embodiment of the present disclosure, a defocusrange (Deforcus_Range) is set according to the distance from the uservisual point position of the user 273, and the content is displayedafter processing the content into a state that the content is defocusedin a degree corresponding to the defocus range.

As a result, even when the user moves the visual line, the content at aposition close to the visual point position EP is displayed in a focusedstate with a small degree of defocus according to the motion of thevisual line. The content at a position away from the visual pointposition EP is displayed in a defocused state according to the size ofthe distance from the visual point position EP. This can realize anatural view according to the visual line direction of the user.

<<2-2. Setting Example of Defocus Range>>

Next, a setting example of the defocus range will be described withreference to FIG. 43 .

The user 273 focuses on the visual point position to watch an object,with the visual point position at the object in the visual linedirection. Therefore, an object existing at a position away from thevisual point position is in a defocused state.

Therefore, the defocus range is set according to a distance differencevalue that is a difference value of the distance between the position onthe content and the visual point position of the user 273.

For example, in a case considered here illustrated on the right side ofFIG. 43 , an information superimposition appropriate frame 281F is seton a side surface of a building 281, and positions 281A, 281B, and 281Care set in ascending order according to the distance from the visualpoint position of the user 273 on the information superimpositionappropriate frame 281F.

Here, FIG. 43 illustrates a setting example of the defocus range, andthe right side of FIG. 43 illustrates an example of a scene in a casewhere there are buildings 281 and 282 on the left and right on the nearside in the traveling direction of the user 273 riding on the vehicle.In addition, the left side of FIG. 43 illustrates a positionalrelationship between the user 273 and buildings 281 and 283 as viewedfrom above in the case where the scene on the right side of FIG. 43 isviewed.

In a case considered here illustrated on the left side of FIG. 43 , thevisual line of the user 273 is toward the position 281B. Hereinafter,the visual point position in the direction of the visual line of theuser 273 will be referred to as a user visual point position. Therefore,as illustrated on the left side of FIG. 43 , the position 281B is theuser visual point position in a case where an arrow of a dotted lineindicates the visual line of the user 273.

In this case, the position 281B on the information superimpositionappropriate frame 281F is the user visual point position, and therefore,it can be considered that the user 273 is watching and focusing on theposition 281B that is the user visual point position.

In other words, it can be considered that the user 273 is watching thescene in the defocused state at the positions 281A and 281C that are notthe user visual point position on the information superimpositionappropriate frame 281F. It can be considered that the farther thedistance from the user visual point position, the more defocused thecontent.

Therefore, as illustrated in FIG. 43 , the defocus range(Deforcus_Range) is set to 0% at the position 281B that is the uservisual point position. In addition, at the positions 281A and 281C awayfrom the user visual point position, the defocus range (Deforcus_Range)is set to 0.1A % to 0.5A % according to the distance difference valuesthat are values of the distances from the user visual point positions tothe positions 281A and 281C.

That is, the larger the distance difference value between the uservisual point position of the user 273 and the position, the larger thevalue set for the defocus range. The smaller the distance differencevalue, the smaller the value set for the defocus range. Therefore, thedistance difference value is 0 at the position 281B that is the uservisual point position, and the defocus range is also set to 0.

In other words, the closer the position from the user visual pointposition, the smaller the value set for the defocus range. The fartherthe position from the user visual point position, the larger the valueset for the defocus range.

Here, A in the setting example of the defocus range in FIG. 43 is apredetermined coefficient which can be arbitrarily set as a weightaccording to the defocus range and the distance difference value. Thatis, in a case where the coefficient A is small, a change in the distancedifference value between the user visual point position and the positiondoes not significantly change the defocus range, and the defocusingprocess for significantly obscurely displaying the scene is not executedat a position near the user visual point position. In contrast, in acase where the coefficient A is large, even a slight change in thedistance difference value significantly changes the defocus range, andthe defocusing process for obscurely displaying the content is executedeven at a position near the user visual point position.

<<2-3. Setting Example of Defocus Ranges in Case where there isPlurality of Information Superimposition Appropriate Frames>>

Although there is one information superimposition appropriate frame inthe setting example of the defocus range described above, the defocusranges can be similarly set in a case where there is a plurality ofinformation superimposition appropriate frames.

For example, in a case considered here illustrated in FIG. 44 , there isa plurality of buildings in the projection unit 106A, and theinformation superimposition appropriate frames are set for the pluralityof buildings.

FIG. 44 illustrates a scene including a road 290 on the front in thetraveling direction of the vehicle. Buildings 291 to 293 are arranged inascending order of a distance from the left side of the road 290, andbuildings 294 and 295 exist in ascending order of a distance from theright side of the road 290.

Furthermore, in FIG. 44 , an information superimposition appropriateframe 291A is set on a surface of the building 291 facing the travelingdirection of the user, and star-shaped content 291 a is superimposed anddisplayed. Here, it is assumed that a user visual point position 301 ofthe user 273 is substantially at the center of the content 291 a.Therefore, the defocus range (Deforcus_Range) is set to 0% for thecontent 291 a of the information superimposition appropriate frame 291A.

Here, the distances from the user visual point position 301 to theimages on the information superimposition appropriate frames projectedby the projection unit 106A and to the buildings 291 to 295 directlyviewed in front in the traveling direction through the projection unit106A are in order of the buildings 291, 292, 293, 294, and 295 inascending order.

In addition, an information superimposition appropriate frame 292A isset on a surface of the building 292 facing the traveling direction ofthe user, and star-shaped content 292 a is superimposed and displayed.The distance from the user visual point position 301 to the building 292is farther than to the building 291, and the defocus range(Deforcus_Range) is set to, for example, 1.0A % to 1.9A % for thecontent 292 a of the information superimposition appropriate frame 292A.

Furthermore, an information superimposition appropriate frame 294A isset on a surface of the building 294 facing the road 290, andstar-shaped content 294 a is superimposed and displayed. The distancefrom the user visual point position 301 to the building 294 is fartherthan to the building 292, and the defocus range (Deforcus_Range) is setto, for example, 2.0A to 2.9A %, which are values larger than the valuesfor the content 292 a, for the content 294 a of the informationsuperimposition appropriate frames 294A.

In addition, an information superimposition appropriate frame 295A isset on a surface of the building 295 facing the traveling direction ofthe user, and star-shaped content 295 a is superimposed and displayed.The distance from the user visual point position 301 to the building 295is farther than to the building 294, and the defocus range(Deforcus_Range) is set to, for example, 3.0A % to 3.9A %, which arevalues larger than the values for the content 294 a, for the content 295a of the information superimposition appropriate frame 295A.

That is, the defocus range of the content 291 a at the user visual pointposition 301 is set to the minimum value of 0%.

In addition, the defocus range of the content 292 a in the informationsuperimposition appropriate frame 292A at the closest distance to theinformation superimposition appropriate frame 291A as the user visualpoint position 301 is set to, for example, 1.0% to 1.9%.

Furthermore, the defocus range of the content 294 a in the informationsuperimposition appropriate frame 294A at the second closest distance tothe information superimposition appropriate frame 291A as the uservisual point position 301 is set to, for example, 2.0% to 2.9%.

In addition, the defocus range of the content 295 a in the informationsuperimposition appropriate frame 295A at the third closest distance tothe information superimposition appropriate frame 291A as the uservisual point position 301 is set to, for example, 3.0% to 3.9%.

Therefore, the content 291 a is displayed in a focused state. Thecontent 292 a is displayed in a defocused state such that the defocusrange corresponds to 1.0% to 1.9%, and the content 292 a is displayedmore obscurely for the user 273 compared to the content 291 a.

In addition, the content 294 a is displayed in a defocused state suchthat the defocus range corresponds to 2.0% to 2.9%, and the content 294a is displayed more obscurely for the user 273 compared to the content292 a.

Furthermore, the content 295 a is displayed in a defocused state suchthat the defocus range corresponds to 3.0% to 3.9%, and the content 295a is displayed more obscurely for the user 273 compared to the content294 a.

That is, the degree of defocus increases in the order of the contents292 a, 294 a, and 295 a, and the contents are more obscurely displayed.In FIG. 44 , star shapes indicated by dotted lines express the defocusedstate. Note that, in FIG. 44 , the content 291 a is displayed in a starshape with only solid lines, and this indicates that there is nodeviation of focus. In addition, the content 292 a is displayed in astar shape indicated by one solid line and one dotted line, and thisexpresses that the content 292 a is more defocused and more obscurelydisplayed than the content 291 a.

Furthermore, the content 294 a is displayed in a star shape indicated byone solid line and two dotted lines, and this expresses that the content294 a is more defocused and more obscurely displayed than the content292 a.

Furthermore, the content 295 a is displayed in a star shape indicated byone solid line and three dotted lines, and this expresses that thecontent 295 a is more defocused and more obscurely displayed than thecontent 294 a.

As a result of the display, the content 291 a at the user visual pointposition 301 is displayed in the focused state, and the contents 292 a,294 a, and 295 a are displayed in the defocused, obscure, and blurredstates according to the distance difference values between the uservisual point position 301 of the user 273 and the respective positions.This can realize natural display corresponding to the visual line.

<<2-4. Configuration Example of Second Embodiment of Display ControlUnit>> <2-4-1. Configuration Example of Display Control Unit>

Next, a configuration example of the second embodiment of the outputcontrol unit 105 will be described with reference to FIG. 45 . However,in the configuration example of the second embodiment of the outputcontrol unit 105, the difference from the configuration example of thefirst embodiment is the configuration of the display control unit 203,and only the configuration of the display control unit 203 will bedescribed.

Furthermore, in the display control unit 203 of FIG. 45 , the process ofthe display processing unit 254 is different from the process of thedisplay control unit 203 in FIG. 33 , and the process of the displayprocessing unit 254 will be described.

The display control unit 203 of FIG. 45 uses the scene image captured bythe camera 271 that images the vehicle exterior, the map data stored inthe storage unit 111 or map data acquired from a server not illustrated,and the information of the information superimposition appropriateframes to construct a pseudo-3D model on the memory space and generates,in the pseudo-3D model, wireframes corresponding to the informationsuperimposition appropriate frames.

In addition, the display control unit 203 uses information of acoordinate system of the pseudo-3D model and the vehicle interior imagecaptured by the camera 272 to obtain the defocus range at each positionon the information superimposition appropriate frames superimposed onthe wireframes in the pseudo-3D model and generates a defocus range map.

Furthermore, the display control unit 203 superimposes the content withadjusted contrast and the part images on the wireframes in the pseudo-3Dmodel and reads the defocus range of each position from the defocusrange map to execute a corresponding defocusing process. The displaycontrol unit 203 outputs the pseudo-3D model to the projection unit 106Aand causes the projection unit 106A to display the pseudo-3D model.

<2-4-2. Configuration Example of Display Processing Unit of FIG. 45>

Next, a detailed configuration example of the display processing unit254 of FIG. 45 will be described with reference to FIG. 46 .

The display processing unit 254 of FIG. 45 includes a pseudo-3D modelwireframe generation unit 321, a content superimposition unit 322, adefocus processing unit 323, a visual point position detection unit 324,a defocus range setting unit 325, and a defocus range map storage unit326.

The pseudo-3D model wireframe generation unit 321 uses the road, thebuilding, and the like corresponding to the space in the travelingdirection of the vehicle to construct a pseudo-3D model in the memoryspace on the basis of the scene image, the map data, and the informationof the information superimposition appropriate frames. Furthermore, thepseudo-3D model wireframe generation unit 321 generates, in theconstructed pseudo-3D model, corresponding wireframes on the basis ofthe information of the positions provided with the informationsuperimposition appropriate frames and outputs the wireframes aspseudo-3D model wireframe information to the content superimpositionunit 322, the visual point position detection unit 324, and the defocusrange setting unit 325.

The content superimposition unit 322 superimposes the content withadjusted contrast and the part images on the wireframes corresponding tothe information superimposition appropriate frames in the pseudo-3Dmodel and outputs the wireframes to the defocus processing unit 323.

The defocus processing unit 323 accesses the defocus range map storageunit 326 and reads the set defocus range group for each wireframecorresponding to the information superimposition appropriate frame inthe pseudo-3D model provided with the content with adjusted contrast andthe part images. The defocus processing unit 323 applies the defocusingprocess to the image in the corresponding defocus range and outputs theimage.

The visual point position detection unit 324 searches for the uservisual point position from the vehicle interior image captured by thecamera 272 and obtains the coordinate position on the 3D modelwireframe. The visual point position detection unit 324 outputs thecoordinate position to the defocus range setting unit 325. Morespecifically, the visual point position detection unit 324 detects theface image of the user from the vehicle interior image captured by thecamera 272. Furthermore, the visual point position detection unit 324specifies the iris positions of the eyes from the detected face imageand detects, as the user visual point position, the visual pointposition on the projection surface of the projection unit 106A on thebasis of the positions of the irises.

The defocus range setting unit 325 sets the defocus range at eachposition in the wireframes corresponding to the informationsuperimposition appropriate frames in the pseudo-3D model on the basisof the pseudo-3D model wireframe information and the information of theuser visual point position to form a defocus range group for eachwireframe and causes the defocus range map storage unit 326 to store thedefocus range groups as a defocus range map regarding the plurality ofinformation superimposition appropriate frames. Furthermore, in thiscase, the defocus range setting unit 325 also sets the defocus rangesfor regions other than the wireframes. That is, the defocus rangesetting unit 325 sets the defocus ranges for all of the positions in thepseudo-3D model space and causes the defocus range map storage unit 326to store the defocus ranges as a defocus range map.

The defocus range map storage unit 326 stores the defocus range map. Thedefocus processing unit 323 applies the defocusing process correspondingto the defocus ranges to the content supplied from the contentsuperimposition unit 322, in which the content with adjusted contrastand the part images are superimposed on the wireframes corresponding tothe information superimposition appropriate frames in the pseudo-3Dmodel space. The defocus processing unit 323 outputs the content.

<2-4-3. Display Process of Display Control Unit in FIG. 45>

Next, the display process of the display control unit 203 in FIG. 45will be described with reference to a flow chart of FIG. 47 .

In step S201, the pseudo-3D model wireframe generation unit 321constructs the pseudo-3D model of the road, the buildings, and the likein the memory space on the basis of the scene image, the map data, andthe information regarding the information superimposition appropriateframes. Furthermore, the pseudo-3D model wireframe generation unit 321sets, in the pseudo-3D model, the wireframes corresponding to theinformation superimposition appropriate frames on the basis of theinformation of the positions provided with the informationsuperimposition appropriate frames and outputs the wireframes as thepseudo-3D model wireframe information to the content superimpositionunit 322, the visual point position detection unit 324, and the defocusrange setting unit 325.

In step S202, the visual point position detection unit 324 and thedefocus range setting unit 325 execute the defocus range map generationprocess to generate the defocus range map and cause the defocus rangemap storage unit 326 to store the defocus range map.

<2-4-4. Defocus Range Map Generation Process of Display Control Unit inFIG. 45>

Here, the defocus range map generation process of the display controlunit 203 in FIG. 45 will be described with reference to a flow chart ofFIG. 48 .

In step S221, the visual point position detection unit 324 acquires thevehicle interior image captured by the camera 272 and detects the faceimage of the user as an occupant. The visual point position detectionunit 324 further detects the iris positions to estimate the visual linedirection on the basis of the iris positions and specifies the uservisual line position on the display surface of the projection unit 106A.

In step S222, the visual point position detection unit 324 specifies thecoordinate position of the user visual point position on the projectionunit 106A in the pseudo-3D model and outputs the coordinate position ofthe user visual point position to the defocus range setting unit 325.

In step S223, the defocus range setting unit 325 sets, for each point inthe pseudo-3D model, the defocus range corresponding to the differencevalue of a distance from the user visual point position to generate thedefocus range map and causes the defocus range map storage unit 326 tostore the defocus range map.

As a result of the process described above, the defocus range at eachpoint of the 3D model is set, and the defocus ranges are stored as thedefocus range map in the defocus range map storage unit 326. In thiscase, continuous defocus ranges are set in the wireframes of the 3Dmodel in the defocus range map, and the defocus range groups are formedon the basis of wireframes. The defocus ranges are stored in the defocusrange map in this state.

Here, the flow chart of FIG. 47 will be further described.

In step S202, once the defocus range map is generated in the defocusrange map generation process, the fitting contrast calculation unit 251of the display control unit 203 focuses on one content provided with thesuperimposition location in step S203.

In step S204, the fitting contrast calculation unit 251 calculates thefitting contrast on the basis of the context at the location of theinformation superimposition appropriate frame as the superimpositionlocation.

In step S205, the contrast adjustment unit 252 adjusts the contrast ofthe focused content on the basis of the fitting contrast.

In step S206, the mask processing unit 253 executes the masking processto cut out the image of the front object in a case where there is afront object in the information superimposition appropriate frame as thesuperimposition location of the focused content.

In step S207, the content superimposition unit 322 of the displayprocessing unit 254 superimposes the content with adjusted contrast onthe wireframe in the pseudo-3D model space corresponding to theinformation superimposition appropriate frame.

In step S208, the content superimposition unit 322 appropriatelydisplays the image of the front object on top of the content and outputsthe content to the defocus processing unit 323.

In step S209, the defocus processing unit 323 accesses the defocus rangemap storage unit 326 and reads, from the defocus range map, the defocusrange group corresponding to the frame provided with the focusedcontent. In accordance with the focus range at each point on the frame,the defocus processing unit 323 applies the defocusing process to eachpoint on the wireframe of the pseudo-3D model corresponding to theinformation superimposition appropriate frame provided with the contentand outputs the frame.

In step S210, the display control unit 203 determines whether or not allof the contents are focused.

In a case where the display control unit 203 determines that there arecontents not focused yet in step S210, the display control unit 203returns to step S203 to focus on another content and repeats the processdescribed above.

In a case where the display control unit 203 determines that all of thecontents are focused in step S210, all of the contents are superimposedon the wireframes of the pseudo-3D model corresponding to theinformation superimposition appropriate frames set as thesuperimposition locations. Once the defocusing process is executed, theprocess proceeds to step S211.

In step S211, the defocus processing unit 323 outputs the imagesubjected to the defocusing process according to the distances from theuser visual point position in the state that all of the contents aresuperimposed on the information superimposition appropriate frames setas the superimposition locations. The defocus processing unit 323outputs the image to the projection unit 106A and causes the projectionunit 106A to display the image. The process returns to step S5 of FIG.36 , and the subsequent process is executed.

As a result of the series of processes described above, on theprojection unit 106A including the transmissive display, the contentsare superimposed and displayed on the information superimpositionappropriate frames such that the contents are fit into the scene. Inthis case, the closer the position of the content of the informationsuperimposition appropriate frame to the user visual point position, thesmaller the degree of defocus in the displayed image (more focused,sharp, and clear image). The farther the position of the content of theinformation superimposition appropriate frame from the user visual pointposition, the larger the degree of defocus in the displayed image (moredefocused and obscure image).

As a result, the content can be displayed at optimal places in the scenein a form that the content is more fit into the scene. The user canwatch the content in a way similar to viewing a normal scene.

Note that, although the defocus range of each point in the 3D modelcorresponding to the entire scene image is obtained to generate thedefocus range map in the defocus range map generation process in theexample described above, the defocusing process is actually applied onlyto the regions of the information superimposition appropriate framesprovided with the content. Therefore, for the defocus range map, onlythe defocus range groups of the regions of the wireframes in thepseudo-3D model corresponding to the regions of the informationsuperimposition appropriate frames may be obtained to form the defocusrange map.

<<2-5. Modification>>

<2-5-1. Configuration Example of Display Processing Unit in Case whereProjection Unit is Non-Transmissive Display>

Although the projection unit 106A in FIG. 42 includes the transmissivedisplay in the example of the case described above, the projection unit106A may include a non-transmissive display.

That is, the projection unit 106A is the transmissive display in thedescription above, and the content is superimposed on the regionsassumed to be the information superimposition appropriate framesdescribed above. Therefore, the content is projected after applying thedefocusing process only to the regions provided with the content thatare assumed to be the information superimposition appropriate frames.Thus, the user watches the projected images in the regions assumed to bethe information superimposition appropriate frames and directly viewsand watches the objects existing on the front side in the regions otherthan the regions assumed to be the information superimpositionappropriate frames. In this way, the user watches the content displayedto naturally fit into the scene.

However, in a case of a non-transmissive display, the entire displaydisplays, for example, the scene image captured by the camera 271, andthe content is superimposed and displayed in the regions of theinformation superimposition appropriate frames in the displayed sceneimage. That is, in the case where the content is displayed in thenon-transmissive display, the user watches the images in the state inwhich the content is displayed on part of the images displayed on theentire display. Therefore, in the case of the non-transmissive display,the defocusing process corresponding to the user visual point positionis necessary in the entire display region of the display.

Here, a configuration example of the display processing unit in the casewhere the projection unit is a non-transmissive display will bedescribed with reference to FIG. 49 .

In the display processing unit 254 of FIG. 49 , the configurationdifferent from the display processing unit 254 of FIG. 46 is that theinformation of the scene image is supplied to the defocus processingunit 323, and the defocus processing unit 323 executes the defocusingprocess in the state in which the information of the informationsuperimposition appropriate frames is superimposed.

That is, in the case where the projection unit 106A is anon-transmissive display, the defocus processing unit 323 applies thedefocusing process to the entire scene image in the state in which theinformation superimposition appropriate frames provided with the contentare superimposed on the scene image captured by the camera 271.

In this case, the defocus processing unit 323 uses all of theinformation of the defocus ranges set in the entire scene image storedin the defocus range map storage unit 326 and applies the defocusingprocess to the entire scene image in the state in which the informationsuperimposition appropriate frames provided with the content aresuperimposed.

<2-5-2. Display Process in Case where Projection Unit isNon-Transmissive Display>

Next, a display process in the case where the projection unit 106A is anon-transmissive display will be described with reference to a flowchart of FIG. 50 . Note that the process of steps S241 to S249 in FIG.49 is similar to the process of steps S201 to S208 and S210 in FIG. 47 ,and the description will not be repeated.

That is, once the content is superimposed on all of the wireframescorresponding to the information superimposition appropriate frames inthe pseudo-3D model in the process of steps S241 to S249, the processproceeds to step S250.

In step S250, the defocus processing unit 323 attaches and superimposesthe wireframes on the scene image captured by the camera 271 in thestate in which the content is superimposed on all of the wireframescorresponding to the information superimposition appropriate frames inthe pseudo-3D model.

In step S251, the defocus processing unit 323 applies the defocusingprocess to the entire scene image according to the defocus ranges. Thatis, here, the defocusing process with the defocus ranges correspondingto the difference values of distances from the user watching position isapplied not only to the regions provided with the content on thewireframes corresponding to the information superimposition appropriateframes, but also to each position of the entire scene image in thepseudo-3D model.

In step S252, the defocus processing unit 323 outputs the scene imagesubjected to the defocusing process to the projection unit 106Aincluding the non-transmissive display and causes the projection unit106A to display the scene image. The process returns to step S5 of FIG.36 , and the subsequent process is executed.

That is, a display example of FIG. 51 illustrates an example in whichthe scene displayed on the projection unit 106A on the basis of thetransmissive display described with reference to FIG. 44 is displayed bya projection unit 106A including a non-transmissive display.

That is, the scene is displayed by the projection unit 106A includingthe transmissive display in the example of FIG. 44 , and therefore, thedefocusing process is applied to only the respective star-shapedcontents 292 a, 294 a, and 295 a in the information superimpositionappropriate frames 292A, 294A, and 295A according to the distance fromthe user visual point position 301.

In contrast, the scene is displayed by the projection unit 106Aincluding the non-transmissive display in the example of FIG. 51 , andthe defocusing process is applied to the entire scene image includingbuildings 292′ to 295′ corresponding to the buildings 292 to 295according to the distance from the user visual point position 301.Therefore, the defocusing process is also applied to each of theinformation superimposition appropriate frames 292A, 294A, and 295Asimilarly to the star-shaped contents 292 a, 294 a, and 295 a. Thedefocusing process is also applied to other configurations. Note that,as for the defocusing process in the star-shaped contents 292 a, 294 a,and 295 a, a similar defocusing process is executed regardless ofwhether the projection unit 106A is a transmissive display or anon-transmissive display.

As a result, the content can be displayed at optimal places in the scenein a form that the content is more fit into the scene, and the user canwatch the content in a way similar to viewing a normal scene.

3. Third Embodiment <<3-1. Summary of Third Embodiment>>

Although the defocusing process is applied to the content on theinformation superimposition appropriate frames according to thedifference values of distances from the user visual point position inthe case where there is one user as an occupant in the example describedabove, defocusing processes may be executed according to respective userwatching positions to allow respective users to watch images subjectedto optimal defocusing processes in a case where there is a plurality ofusers as occupants.

FIG. 52 is a diagram describing a summary of a third embodiment of thevehicle control system 100 that defocuses the content superimposed onthe information superimposition appropriate frames according torespective visual point positions (visual line directions) of aplurality of users riding on a vehicle and that displays the content toallow respective users to appropriately watch the content. Note that, inFIG. 52 , the description of the components with the same functions asthe components in FIG. 42 will be appropriately omitted.

That is, FIG. 51 is different from FIG. 42 in that the users asoccupants include three users 273-1 to 273-3, cameras 272 include threecameras 272-1 to 272-3 accordingly, and a display unit 351 is providedin place of the projection unit 106A.

The cameras 272-1 to 272-3 image the vehicle interior to image the users273-1 to 273-3 as occupants, respectively, and supply the capturedvehicle interior images to the output control unit 105. In this case, asin the case where there is one occupant, the output control unit 105specifies a user visual point position EP1 of the user 273-1 from theimage captured by the camera 272-1, specifies a user visual pointposition EP2 of the user 273-2 from the image captured by the camera272-2, and specifies a user visual point position EP3 of the user 273-3from the image captured by the camera 272-3. Note that only one camera272 may be included, and the camera 272 may be able to image the entirevehicle interior. In this way, a plurality of users may be detected fromone image, and the respective user visual point positions EP1 to EP3 maybe detected.

In addition, on the basis of the information of the respective uservisual point positions EP1 to EP3 of the users 273-1 to 273-3, theoutput control unit 105 generates scene images that allow the respectiveusers 273-1 to 273-3 to watch the content in a state in which thecontent is naturally superimposed on the information superimpositionappropriate frames. The output control unit 105 integrates the sceneimages into one image in chronological order.

Furthermore, the output control unit 105 outputs the scene imagesintegrated into one image in chronological order to the display unit 351while changing the scene images in chronological order.

The display unit 351 is provided with a liquid crystal polarizationshutter 361 in addition to the projection unit 106A described above, andin displaying the images integrated into one scene image inchronological order, the output control unit 105 switches thepolarization direction of the light transmitted by the liquid crystalpolarization shutter 361 according to the timing of the display of theimage to be watched by each of the users 273-1 to 273-3. In this way,the scene image is displayed in the state that allows each of the users273-1 to 273-3 to watch only the images appropriate for each user.

Here, the configuration of the display unit 351 will be described withreference to FIG. 53 . FIG. 53 illustrates the configuration in the casewhere the projection unit 106A included in the display unit 351includes, for example, the transmissive display, and FIG. 53 is a topview from vertically above the display surface of the display unit 351.

In the case of FIG. 53 , the users 273-1 to 273-3 on the lower side ofFIG. 53 view, through the projection unit 106A, the scene on the upperside of FIG. 53 that is forward in the traveling direction. Therefore,the users 273-1 to 273-3 can watch forward in the traveling direction inthe state in which the content is superimposed on the informationsuperimposition appropriate frames on the projection unit 106A. In thisway, the users 273-1 to 273-3 can watch content such that the content isnaturally fit into the scene.

Meanwhile, when the images including the content that is suitable foreach of a plurality of persons and that is superimposed on theinformation superimposition appropriate frames generated for each of theplurality of persons are integrated into one image in chronologicalorder, the image is displayed as illustrated on the upper side of FIG.53 . That is, for example, an image for the user 273-1 indicated byVideo_01 is displayed at time t1, and an image for the user 273-2indicated by Video_02 is displayed at time t2. An image for the user273-3 indicated by Video_03 is displayed at time t3, and such display isrepeated.

Therefore, when one image integrated in chronological order is displayedin a state without the liquid crystal polarization shutter 361, each ofthe users 273-1 to 273-3 watches images not suitable for watching, twoout of three times.

Therefore, the liquid crystal polarization shutter 361 is provided in aprevious stage of the projection unit 106A, and the output control unit105 controls the integrated image according to the timing of display toswitch the polarization direction.

That is, the output control unit 105 causes the projection unit 106A todisplay the image for the user 273-1 at the timing of time t1 andcontrols the liquid crystal polarization shutter 361 as indicated by adotted line to polarize the light in a direction that allows only theuser 273-1 to watch the image as indicated by arrows of dotted lines.

In addition, the output control unit 105 causes the projection unit 106Ato display the image for the user 273-2 at the timing of time t2 andcontrols the liquid crystal polarization shutter 361 as indicated by asolid line to polarize the light in a direction that allows only theuser 273-2 to watch the image as indicated by arrows of solid lines.

Furthermore, the output control unit 105 causes the projection unit 106Ato display the image for the user 273-3 at the timing of time t3 andcontrols the liquid crystal polarization shutter 361 as indicated by achain line to polarize the light in a direction that allows only theuser 273-3 to watch the image as indicated by arrows of chain lines.

Subsequently, the output control unit 105 repeats similar control tocontrol the display unit 351 to allow each of the users 273-1 to 273-3to watch only the images suitable for the user to watch. Note that thedotted line, the solid line, and the chain line in the liquid crystalpolarization shutter 361 in FIG. 53 express control states of the liquidcrystal polarization shutter 361, and the lines are not indicative ofdifferent shutters or the like.

According to the configuration described above, even when there is aplurality of users riding on the vehicle, the content can be displayedat optimal places in the scene in a state suitable for each user, in aform that the content is more fit into the scene. Each user can watchthe content at each position in a way similar to viewing a normal scene.

<<3-2. Configuration Example of Vehicle Control System>> <3-2-1. OverallConfiguration of Vehicle Control System>

FIG. 54 is a block diagram illustrating a configuration example of thevehicle control system 100 according to the third embodiment. Note that,in the vehicle control system 100 of FIG. 54 , the same reference signsare provided to the components with the same functions as in the vehiclecontrol system 100 of FIG. 8 , and the description thereof will beappropriately omitted.

That is, the vehicle control system 100 of FIG. 54 is different from thevehicle control system 100 of FIG. 8 in that the display unit 351 isprovided in place of the projection unit 106A.

The display unit 351 includes the projection unit 106A and the liquidcrystal polarization shutter 361 as described with reference to FIG. 53.

The liquid crystal polarization shutter 361 switches the polarizationdirection to allow only the user 273 suitable for watching the image towatch the image according to the timing of display of the image when theimages provided with the content on the information superimpositionappropriate frames in the states respectively suitable for the pluralityof users 273 are displayed as one image in chronological order.

Note that, although the projection unit 106A may be a transmissivedisplay or a non-transmissive display, the configuration and the processsuitable for each display are necessary as described in the secondembodiment. That is, in the case where the projection unit 106A is atransmissive display, the display process as illustrated in the flowchart of FIG. 47 needs to be executed in the configuration of thedisplay processing unit 254 as illustrated in FIG. 46 . Furthermore, inthe case where the projection unit 106A is a non-transmissive display,the display process as illustrated in the flow chart of FIG. 50 needs tobe executed in the configuration of the display processing unit 254 asillustrated in FIG. 49 .

<3-2-2. Configuration of Display Processing Unit>

Next, the configuration of the display processing unit 254 in the outputcontrol unit 105 of the vehicle control system 100 of FIG. 54 will bedescribed with reference to FIG. 55 . Note that, in the displayprocessing unit 254 of FIG. 54 , the same reference signs are providedto the components with the same functions as in the display processingunit 254 of FIG. 46 , and the description thereof will be appropriatelyomitted.

That is, the difference from the display processing unit 254 of FIG. 46is that the display processing unit 254 of FIG. 55 newly includes abuffer 371, a combining unit 372, a timing control unit 373, a shuttercontrol unit 374, and a number-of-people detection unit 324 a thatdetects the number of users in the visual point position detection unit324.

The buffer 371 buffers the images, which are generated by using thecomponents from the pseudo-3D model wireframe generation unit 321 to thedefocus range map storage unit 326, generated according to each of theplurality of users 273 watching the images, provided with the content inthe information superimposition appropriate frames, and subjected to thedefocusing process according to the user watching position.

The combining unit 372 integrates and combines, in chronological order,the images, which are buffered by the buffer 371, suitable for each ofthe plurality of users watching the images, provided with the content inthe information superimposition appropriate frames, and subjected to thedefocusing process according to the user watching position, and outputsthe images to the timing control unit 373. Note that the combining unit372 integrates the images into one image in chronological order byincluding information for identifying the timing of the image to bedisplayed and the user 273 suitable for watching the image.

When the images integrated in chronological order are sequentiallysupplied, the timing control unit 373 recognizes any of the users 273suitable for watching the images and causes the projection unit 106A tosequentially display the images. The timing control unit 373 alsocontrols the shutter control unit 374 to control the polarizationdirection of the liquid crystal polarization shutter 361 to thedirection facing the corresponding user 273.

The number-of-people detection unit 324 a uses the vehicle interiorimage to detect the number of users on the basis of, for example, thenumber of face images detected by face detection or the like andnotifies the defocus processing unit 323 of the number of users.Therefore, the defocus processing unit 323 applies the defocusingprocess to the images according to the supplied number of users.

Note that, although FIG. 55 illustrates the configuration of the casewhere the projection unit 106A is a transmissive display, the sceneimage is input to the defocus processing unit 323 as in the displayprocessing unit 254 of FIG. 49 in the case where the projection unit106A is a non-transmissive display. Furthermore, the defocus processingunit 323 attaches, to the scene image, the images provided with thecontent in the information superimposition appropriate frames and thenapplies the defocusing process to the entire scene image. Here, theconfiguration diagram of the display processing unit 254 correspondingto the non-transmissive display will not be illustrated.

<<3-3. Display Process of Display Processing Unit in FIG. 55>>

Next, the display process of the display processing unit 254 of FIG. 55will be described with reference to a flow chart of FIG. 56 .

In step S291, the defocus processing unit 323 initializes a counter m ofan identifier for identifying the user to 1.

In step S292, the number-of-people detection unit 324 a detects thenumber of users M as occupants from the images captured by the cameras272-1 to 272-3, sets the identifier m for each user, and supplies thedetection result to the defocus processing unit 323.

In step S293, the display process of the user identified by theidentifier m among the users 273 is executed.

Here, the display process of the user m is, for example, the process ofsteps S201 to 210 in FIG. 47 corresponding to a user 273-m watching theimage in the case where the projection unit 106A is the transmissivedisplay. The display process of the user m is a process of steps S241 to251 in FIG. 50 corresponding to a user m watching the image in the casewhere the projection unit 106A is the non-transmissive display. That is,in the process, the defocusing process corresponding to the user visualpoint position of the user 273-m is applied to generate the image in thestate in which the content is superimposed on the informationsuperimposition appropriate frames corresponding to the user 273-mwatching the image.

In step S294, the defocus processing unit 323 associates the imagegenerated in the display process of the user m with the identifier m andcauses the buffer 371 to buffer the image.

In step S295, the defocus processing unit 323 determines whether or notthe counter m matches the number of people M, that is, whether or notthe images corresponding to all of the users are generated.

In a case where the counter m does not match M in step S295, that is, ina case where the images corresponding to all of the users are notgenerated, the process proceeds to step S296.

In step S296, the defocus processing unit 323 increments the counter mby 1, and the process returns to step S293.

That is, the process of steps S293 to S296 is repeated until the imagesare buffered by the buffer 371 after generating the images by executingthe display process for all of the users.

Furthermore, in a case where the counter m matches M after generatingthe images corresponding to all of the users in step S295, that is, in acase where it is assumed that the display process is executed for all ofthe users, the process proceeds to step S297.

In step S297, the combining unit 372 integrates the images, which arebuffered by the buffer 371, provided with the content on the informationsuperimposition appropriate frames corresponding to all of the users,and subjected to the defocusing process according to the defocus rangecorresponding to the user watching position, into one image inchronological order.

In step S298, the timing control unit 373 sequentially supplies theintegrated images to the projection unit 106A to cause the projectionunit 106A to display the images and controls the shutter control unit374 at a corresponding timing to control the liquid crystal polarizationshutter 361 to polarize the light in the direction that allows thetarget user 273 to watch the displayed image.

As a result of the process described above, even when a plurality ofusers is riding on the vehicle, the content can be displayed at optimalplaces in the scene in the state suitable for each user, in a form thatthe content is more fit into the scene. Each user can watch the contentat each position in a way similar to viewing a normal scene.

<<3-4. Modification>>

In the description above, the liquid crystal polarization shutter 361switches the polarization directions of the images projected on theprojection unit 106A to realize the plurality of users watching theimages. Instead of using the liquid crystal polarization shutter 361,the images suitable for the plurality of users watching the images,respectively, may be vertically divided in pixel order in the frame tocombine the images into one image, and the images may be displayedthrough lenticular lenses to allow each of the plurality of users towatch the image.

FIG. 57 is a diagram illustrating a configuration example of the displayunit 351 according to a modification. Note that the same reference signsare provided to the components with the same functions as in the displayunit 351 of FIG. 53 , and the description thereof will be appropriatelyomitted.

That is, the display unit 351 of FIG. 57 is different from the displayunit 351 of FIG. 53 in that lenticular lenses 381 are provided in placeof the liquid crystal polarization shutter 361.

Furthermore, as illustrated on the upper right in FIG. 57 , the imageVideo_01 for the user 273-1, the image Video_02 for the user 273-2, andthe image Video_03 for the user 273-3 are repeatedly arranged in a statein which the images are divided into regions C1, C2, and C3 divided onthe basis of columns formed in pixel order to thereby display the imagescombined into one image in the projection unit 106A.

The lenticular lenses 381 selectively transmit the image Video_01 in thedirection that allows the user 273-1 to watch, selectively transmit theimage Video_02 in the direction that allows the user 273-2 to watch, andselectively transmit the image Video_03 in the direction that allows theuser 273-3 to watch, among the images displayed on the projection unit106A.

According to the configuration described above, even when a plurality ofusers is riding on the vehicle, the content can be displayed at optimalplaces in the scene in the state suitable for each user, in a form thatthe content is more fit into the scene. Each user can watch the contentat each position in a way similar to viewing a normal scene.

<3-4-1. Configuration of Display Processing Unit>

Next, the configuration of the display processing unit 254 in the outputcontrol unit 105 of the vehicle control system 100 of FIG. 52 will bedescribed with reference to FIG. 58 . Note that, in the displayprocessing unit 254 of FIG. 57 , the same reference signs are providedto the components with the same functions as in the display processingunit 254 of FIG. 55 , and the description will be appropriately omitted.

That is, the display processing unit 254 of FIG. 58 is different fromthe display processing unit 254 of FIG. 55 in that a combining unit 391is provided in place of the combining unit 372, the timing control unit373, and the shutter control unit 374.

The combining unit 391 divides the images that are buffered by thebuffer 371 and that are respectively suitable for a plurality of userswatching the images into regions on the basis of columns in pixel orderand combines the images into one image. The combining unit 391 outputsthe image to the projection unit 106A of the display unit 351.

The projection unit 106A projects the image obtained by integrating,into one image, the images that are divided into regions on the basis ofcolumns in pixel order and that are respectively suitable for theplurality of users watching the images. In this way, the projection unit106A projects the images through the lenticular lenses 381 in the statethat allows each of the users 273 to appropriately watch the images.

Note that, although the projection unit 106A may be a transmissivedisplay or a non-transmissive display, the configuration and the processsuitable for each display are necessary as described in the secondembodiment. That is, in the case where the projection unit 106A is atransmissive display, the display process as illustrated in the flowchart of FIG. 47 needs to be executed in the configuration of thedisplay processing unit 254 as illustrated in FIG. 46 . Furthermore, inthe case where the projection unit 106A is a non-transmissive display,the display process as illustrated in the flow chart of FIG. 50 needs tobe executed in the configuration of the display processing unit 254 asillustrated in FIG. 49 .

<3-4-2. Display Process of Display Processing Unit in FIG. 57>

Next, the display process of the display processing unit 254 in FIG. 58will be described with reference to a flow chart of FIG. 59 .

The process of steps S311 to S316 in the flow chart of FIG. 59 issimilar to the process of steps S291 to S296 in the flow chart of FIG.56 , and the description thereof will not be repeated.

That is, in the process of steps S311 to S316, the display processcorresponding to all of the users is executed, the content issuperimposed on the information superimposition appropriate framescorresponding to all of the users, images are generated by executing thedefocusing process according to the distances from the respective uservisual point positions, and the images are buffered by the buffer 371.The process then proceeds to step S317.

In step S317, the combining unit 391 divides the images, which arebuffered by the buffer 371, provided with the content on the informationsuperimposition appropriate frames corresponding to all of the users,and subjected to the defocusing process in the defocusing rangescorresponding to the distances from the user watching positions, intoregions on the basis of columns in predetermined pixel order. Thecombining unit 391 combines and integrates the images into one image asdescribed with reference to FIG. 57 .

In step S318, the combining unit 391 causes the projection unit 106A toproject the image obtained by integrating, into one image, the imagessubjected to the defocusing process corresponding to the plurality ofusers and causes the projection unit 106A to selectively polarize thelight through the lenticular lenses 381 and project the image. In thisway, the plurality of users watches the images suitable for therespective users.

As a result of the process described above, even when a plurality ofusers is riding on the vehicle, the content can be displayed at optimalplaces in the scene image in the state suitable for each user, in a formthat the content is more fit into the scene. Each user can watch thecontent at each position in a way similar to viewing a normal scene.

Note that, hereinafter, the process of the case where there is one useras in the second embodiment will be referred to as a single modeprocess, and the process of the case where there is a plurality of usersas in the third embodiment will be referred to as a multi-mode process.

As described above, even just one camera 272 that captures the vehicleinterior image can realize the multi-mode. Therefore, the single modeprocess and the multi-mode process may be switched and used by using onecamera 272.

In addition, although the process of displaying the image naturally fitinto the scene in the traveling direction is mainly described above, thetechnique may also be used in a performance for prompting to watch aspecific object.

That is, even in a state in which there is a plurality of users and themulti-mode process is executed, the control may be performed to allowall of the users to watch the same image according to the visual pointposition of a specific user.

For example, in a situation where a bus guide of a tourist bus or thelike introduces a specific historical or famous spot, the defocusingprocess may be applied to the content superimposed on the informationsuperimposition appropriate frames according to the user visual pointposition of the bus guide as a specific user, and all of the passengersof the bus that are users other than the specific user may be able towatch the image.

This allows a performance for naturally directing the visual line towardthe historical or famous spot introduced by the bus guide now, and theguidance by the bus guide can be easily understood.

Furthermore, in stage production or the like, when the visual line istoward a thing that the stage director particularly wants the user toview, the image is defocused to allow the user to sharply watch thething at the user visual point position that the stage directorparticularly wants the user to view. For other things, the images aremore defocused to obscurely display the images, and the entire audienceof the stage can watch the images. In this way, the technique can alsobe used for stage production and the like.

That is, in this case, the defocus range is set small for the thing thatthe stage director wants the user to watch, and a more focused sharpimage can be recognized. In contrast, the defocus range is set large fora thing that the stage director does not want the user to watch, and theimage can be watched more obscurely. As a result, the effect of thestage production can be increased.

Note that, although the number of users is three in the case describedabove, the number of users may be other numbers.

<Configuration Example of Computer>

The series of processes described above can be executed by hardware orcan be executed by software. In a case where the series of processes areexecuted by software, a program included in the software is installedfrom a program recording medium to a computer incorporated intodedicated hardware, a general-purpose computer, or the like.

FIG. 60 is a block diagram illustrating a configuration example ofhardware of a computer that uses a program to execute the series ofprocesses. The computer illustrated in FIG. 60 executes a predeterminedprogram to realize the functions of the components including theautomatic drive control unit 112 illustrated in FIGS. 8 and 54 .

A CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, anda RAM (Random Access Memory) 1003 are connected to each other through abus 1004.

An input-output interface 1005 is further connected to the bus 1004. Aninput unit 1006 including a keyboard, a mouse, or the like and an outputunit 1007 including a display, a speaker, or the like are connected tothe input-output interface 1005. In addition, a storage unit 1008including a hard disk, a non-volatile memory, or the like, acommunication unit 1009 including a network interface or the like, and adrive 1010 that drives a removable recording medium 1011 are connectedto the input-output interface 1005.

In the computer configured in this way, the CPU 1001 loads the programstored in, for example, the storage unit 1008 to the RAM 1003 throughthe input-output interface 1005 and the bus 1004 to execute the programto thereby execute the series of processes described above.

The program executed by the CPU 1001 is provided by, for example,recording the program in the removable recording medium 1011 or isprovided through a wired or wireless transmission medium, such as alocal area network, the Internet, and digital broadcasting, and theprogram is installed on the storage unit 1008.

Note that the program executed by the computer may be a program forexecuting the processes in chronological order described in the presentspecification or may be a program for executing the processes inparallel or at a necessary timing such as when the processes areinvoked.

The system in the present specification denotes a set of a plurality ofconstituent elements (apparatuses, modules (components), and the like),and whether or not all of the constituent elements are in the samehousing does not matter. Therefore, a plurality of apparatuses stored inseparate housings and connected through a network and one apparatusstoring a plurality of modules in one housing are both systems.

Note that the advantageous effects described in the presentspecification are illustrative only, and the advantageous effects arenot limited. There may also be other advantageous effects.

The embodiments of the present technique are not limited to theembodiments described above, and various changes can be made withoutdeparting from the scope of the present technique.

For example, the present technique can be provided as cloud computing inwhich a plurality of apparatuses shares one function and cooperates toexecute a process through a network.

In addition, one apparatus can execute each step described in the flowcharts, or a plurality of apparatuses can take charge and execute eachstep.

Furthermore, in the case where one step includes a plurality ofprocesses, one apparatus can execute the plurality of processes includedin one step, or a plurality of apparatuses can take charge and executethe processes.

<Example of Combination of Configurations>

The present technique can also be configured as follows.

(1)

An information processing apparatus including:

a setting unit that sets a frame as a superimposition location ofcontent in a region corresponding to a surface of an object on the basisof a movement state of a user; and

a display control unit that generates visual information for displayingthe content in the region corresponding to the set frame.

(2)

The information processing apparatus according to (1), furtherincluding:

a prediction unit that predicts the movement state including a travelingroute and a traveling speed of a moving body on which the user rides.

(3)

The information processing apparatus according to (1) or (2), furtherincluding:

a detection unit that detects the object by analyzing an image obtainedby imaging a scene in a traveling direction.

(4)

The information processing apparatus according to any one of (1) to (3),in which

the setting unit sets the frame on the basis of angles toward thesurface of the object from respective positions including a firstposition on a movement route and a second position after a lapse ofpredetermined time.

(5)

The information processing apparatus according to any one of (1) to (4),in which

the setting unit sets the frame on the basis of an angle of the surfaceof the object with respect to a traveling direction.

(6)

The information processing apparatus according to any one of (1) to (5),in which

the setting unit sets the frame on the basis of an exposed area of thesurface of the object in a case where the surface of the object isblocked by another object in front.

(7)

The information processing apparatus according to any one of (1) to (6),in which

the setting unit sets the frame on the basis of time that the surface ofthe object is included in a visual field of the user obtained on thebasis of the movement state.

(8)

The information processing apparatus according to any one of (1) to (7),further including:

a light state detection unit that detects a state of light on thesurface of the object, in which

the setting unit sets the frame on the basis of the state of light.

(9)

The information processing apparatus according to any one of (1) to (8),further including:

a calculation unit that calculates fitness of each content and eachframe on the basis of information regarding specifications required fordisplaying the content, in which

the display control unit generates the visual information for displayingeach content in the region corresponding to the frame selected on thebasis of the fitness.

(10)

The information processing apparatus according to (9), in which

the calculation unit calculates the fitness on the basis of an elementthat varies according to a type of the content.

(11)

The information processing apparatus according to (10), in which

in a case where the content is an image, the calculation unit calculatesthe fitness on the basis of a relationship between an aspect ratio ofthe image and an aspect ratio of the frame.

(12)

The information processing apparatus according to (11), in which

in a case where the content is a moving image, the calculation unitcalculates the fitness also on the basis of a relationship betweenreproduction time of the moving image and time that the frame isincluded in a visual field of the user.

(13)

The information processing apparatus according to (10), in which

in a case where the content includes characters, the calculation unitcalculates the fitness on the basis of at least any one of arelationship between an aspect ratio of a display range of thecharacters and an aspect ratio of the frame and a relationship betweenviewing time defined by the number of characters and the time that theframe is included in a visual field of the user.

(14)

The information processing apparatus according to any one of (1) to(13), in which

the display control unit applies a defocusing process to the visualinformation that is generated in the region corresponding to the setframe and is for displaying the content on the basis of a visual pointposition of the user riding on a moving body.

(15)

The information processing apparatus according to (14), furtherincluding:

an imaging unit that images the user riding on the moving body; and

a defocus range setting unit that sets a defocus range according to adistance between the specified visual point position of the user andeach position of the region corresponding to the frame on the basis ofan image taken by the imaging unit, in which

the display control unit applies, according to the defocus range, thedefocusing process to the visual information that is generated in theregion corresponding to the set frame and is for displaying the contenton the basis of the visual point position of the user riding on themoving body.

(16)

The information processing apparatus according to (15), in which

the closer the distance between each position of the regioncorresponding to the frame and the visual point position of the user is,the smaller the defocus range set by the defocus range setting unit is,the farther the distance between each position of the regioncorresponding to the frame and the visual point position of the user is,the larger the defocus range set by the defocus range setting unit is,and

the smaller the defocus range is, the higher a degree of focus in thedefocusing process of the display control unit applied to the visualinformation that is generated in the region corresponding to the setframe and is for displaying the content on the basis of the visual pointposition of the user riding on the moving body is, the larger thedefocus range is, the higher a degree of defocus in the defocusingprocess of the display control unit is.

(17)

The information processing apparatus according to any one of (14) to(16), in which

the setting unit sets, for each of a plurality of the users, the frameas the superimposition location of the content in the regioncorresponding to the surface of the object on the basis of the movementstate of the user, and

the display control unit applies the defocusing process to the visualinformation that is generated in the region corresponding to the frameset for each of the plurality of the users and is for displaying thecontent.

(18)

An information processing method executed by an information processingapparatus, the method including:

setting a frame as a superimposition location of content in a regioncorresponding to a surface of an object on the basis of a movement stateof a user; and

generating visual information for displaying the content in the regioncorresponding to the set frame.

(19)

A program for causing a computer to execute a process including:

setting a frame as a superimposition location of content in a regioncorresponding to a surface of an object on the basis of a movement stateof a user; and

generating visual information for displaying the content in the regioncorresponding to the set frame.

(20)

A moving body including:

a setting unit that sets a frame as a superimposition location ofcontent in a region corresponding to a surface of an object on the basisof a movement state of a user;

a display control unit that generates visual information for displayingthe content in the region corresponding to the set frame; and

an output unit that displays the visual information.

(21)

The information processing apparatus according to any one of (1) to(13), further including:

an adjustment unit that adjusts contrast of the content on the basis ofcontext of the frame, in which

the display control unit generates, in the region corresponding to theframe, the visual information for displaying the content with adjustedcontrast.

(22)

The information processing apparatus according to (21), in which

the adjustment unit uses, as the context, at least any one of a state ofsunlight, a state of lighting, a state of atmosphere, and the distanceto the frame to adjust the contrast of the content.

(23)

The information processing apparatus according to (21) or (22), furtherincluding:

an image processing unit that cuts out, from the image obtained byimaging, a region of another object in front of the frame, in which

the display control unit generates the visual information for displayingan image of the region of the other object on top of the content withadjusted contrast.

REFERENCE SIGNS LIST

-   -   105 Output control unit, 201 Information superimposition        appropriate visual field setting unit, 202 Superimposition        target frame selection unit, 203 Display control unit, 204        Content acquisition unit, 211 Image analysis unit, 212 Light        state mode setting unit, 213 Object detection unit, 214 Frame        setting unit, 215 Appropriate visual field setting unit, 231        Pre-analysis unit, 232 Fitness calculation unit, 233        Superimposition location setting unit, 251 Fitting contrast        calculation unit, 252 Contrast adjustment unit, 253 Mask        processing unit, 254 Display processing unit, 271, 272, 272-1 to        272-3 Camera, 321 Pseudo-3D model wireframe generation unit, 322        Content superimposition unit, 323 Defocus processing unit, 324        Visual point position detection unit, 324 a Number-of-people        detection unit, 325 Defocus range setting unit, 326 Defocus        range map storage unit, 361 Liquid crystal polarization shutter,        371 Buffer, 372 Combining unit, 373 Timing control unit, 374        Shutter control unit, 381 Lenticular lens, 391 Combining unit

1. (canceled) 2: An information processing apparatus comprising:circuitry configured to: determine a superimposition location of contentin a frame based on a visual field of view relative to a surface of anobject, wherein the visual field of view includes an externalenvironment of a vehicle; perform a defocusing process on the content inthe frame to generate a defocused content; and generate visualinformation for displaying the defocused content on a display. 3: Theinformation processing apparatus according to claim 2, wherein thecircuitry is configured to perform the defocusing process on the contentbased on a visual point position of a user. 4: The informationprocessing apparatus according to claim 3, wherein the circuitry isconfigured to set a defocus range based on a distance between the visualpoint position of the user and the superimposition location of thecontent in the frame, and the circuitry is configured to perform thedefocusing process on the content based on the set defocus range. 5: Theinformation processing apparatus according to claim 4, wherein thesmaller the distance between the visual point position of the user andthe superimposition location of the content is, the smaller the defocusrange set by the circuitry is, and the bigger the distance is, thelarger the defocus range set by the circuitry is, and the smaller thedefocus range is, the higher a degree of focus in the defocusingprocess, and the larger the defocus range is, the higher a degree ofdefocus in the defocusing process. 6: The information processingapparatus according to claim 4, wherein the smaller the distance betweenthe visual point position of the user and the superimposition locationof the content is, the smaller the defocus range set by the circuitryis, and the bigger the distance is, the larger the defocus range set bythe circuitry is. 7: The information processing apparatus according toclaim 4, wherein the smaller the defocus range is, the higher a degreeof focus in the defocusing process, and the larger the defocus range is,the higher a degree of defocus in the defocusing process. 8: Theinformation processing apparatus according to claim 3, wherein thevisual point position of the user is determined based on an imageincluding an eye of the user, the image being generated by a cameramounted inside the vehicle. 9: The information processing apparatusaccording to claim 4, wherein the farther the distance between thevisual point position of the user and the superimposition location ofthe content is, the more obscure a view of the content is. 10: Aninformation processing method executed by an information processingapparatus, the method comprising: determining a superimposition locationof content in a frame based on a visual field of view relative to asurface of an object, wherein the visual field of view includes anexternal environment of a vehicle; performing a defocusing process onthe content in the frame to generate a defocused content; and generatingvisual information for displaying the defocused content on a display.11. The information processing method of claim 10, further comprising:performing the defocusing process on the content based on a visual pointposition of a user.
 12. The information processing method of claim 11,further comprising: setting a defocus range based on a distance betweenthe visual point position of the user and the superimposition locationof the content in the frame; and performing the defocusing process onthe content based on the set defocus range.
 13. The informationprocessing method of claim 12, wherein the smaller the distance betweenthe visual point position of the user and the superimposition locationof the content is, the smaller the defocus range is set, and the biggerthe distance is, the larger the defocus range is set, and the smallerthe defocus range is, the higher a degree of focus in the defocusingprocess, and the larger the defocus range is, the higher a degree ofdefocus in the defocusing process.
 14. The information processing methodof claim 12, wherein the smaller the distance between the visual pointposition of the user and the superimposition location of the content is,the smaller the defocus range set by the circuitry is, and the biggerthe distance is, the larger the defocus range set by the circuitry is.15. The information processing method of claim 12, wherein the smallerthe defocus range is, the higher a degree of focus in the defocusingprocess, and the larger the defocus range is, the higher a degree ofdefocus in the defocusing process.
 16. A non-transitorycomputer-readable storage medium storing executable instructions whichwhen executed by circuitry cause the circuitry to perform a method, themethod comprising: determining a superimposition location of content ina frame based on a visual field of view relative to a surface of anobject, wherein the visual field of view includes an externalenvironment of a vehicle; performing a defocusing process on the contentin the frame to generate a defocused content; and generating visualinformation for displaying the defocused content on a display.
 17. Thenon-transitory computer-readable storage medium of claim 16, furthercomprising: performing the defocusing process on the content based on avisual point position of a user.
 18. The non-transitorycomputer-readable storage medium of claim 17, further comprising:setting a defocus range based on a distance between the visual pointposition of the user and the superimposition location of the content inthe frame; and performing the defocusing process on the content based onthe set defocus range.
 19. The non-transitory computer-readable storagemedium of claim 18, wherein the smaller the distance between the visualpoint position of the user and the superimposition location of thecontent is, the smaller the defocus range is set, and the bigger thedistance is, the larger the defocus range is set, and the smaller thedefocus range is, the higher a degree of focus in the defocusingprocess, and the larger the defocus range is, the higher a degree ofdefocus in the defocusing process.
 20. The non-transitorycomputer-readable storage medium of claim 18, wherein the smaller thedistance between the visual point position of the user and thesuperimposition location of the content is, the smaller the defocusrange set by the circuitry is, and the bigger the distance is, thelarger the defocus range set by the circuitry is.
 21. The non-transitorycomputer-readable storage medium of claim 18, wherein the smaller thedefocus range is, the higher a degree of focus in the defocusingprocess, and the larger the defocus range is, the higher a degree ofdefocus in the defocusing process.